Description

assignee = None
closed_at = None
created_at = <Date 2015-11-27.15:50:58.768>
labels = ['expert-regex', 'easy', '3.9', '3.10', '3.11', 'type-feature', 'docs']
title = '[doc] Clarify exactly what \\w matches in UNICODE mode'
updated_at = <Date 2022-02-28.07:39:46.535>
user = 'https://bugs.python.org/zwol'

activity = <Date 2022-02-28.07:39:46.535>
actor = 'slateny'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Regular Expressions']
creation = <Date 2015-11-27.15:50:58.768>
creator = 'zwol'
dependencies = []
files = []
hgrepos = []
issue_num = 25743
keywords = ['easy']
message_count = 5.0
messages = ['255463', '255464', '255465', '407440', '414180']
nosy_count = 7.0
nosy_names = ['ezio.melotti', 'mrabarnett', 'docs@python', 'zwol', 'Andi McClure', 'iritkatriel', 'slateny']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25743'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

1. zwol commented on Nov 27, 2015

   

   zwolmannequin

   on Nov 27, 2015

   MannequinAuthor

   More actions

   The re module documentation does not do a good job of explaining exactly what \w matches. Quoting https://docs.python.org/3.5/library/re.html :

   \w
   For Unicode (str) patterns:
   Matches Unicode word characters; this includes most characters
   that can be part of a word in any language, as well as numbers
   and the underscore.

   Empirically, this appears to mean "everything in Unicode general categories L* and N*, plus U+005F (underscore)". That is a perfectly sensible definition and the documentation should state it in those terms. "Unicode word characters" could mean any number of different things; note for instance that UTS#18 gives a very different definition.

   (Further reading: https://gist.github.com/zackw/3077f387591376c7bf67 plus links therefrom).
2. zwol

   assigned

   docspython

   on Nov 27, 2015
3. zwol

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   on Nov 27, 2015
4. AndiMcClure commented on Nov 27, 2015

   

   AndiMcCluremannequin

   on Nov 27, 2015

   Mannequin

   More actions

   I would like to request also a clear explanation be given for the documentation in the 2.7 branch. From https://docs.python.org/2.7/library/re.html :

   "\w ... If UNICODE is set, this will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database"

   This is ambiguous. Does it mean the "Alphabetic" property from UAX#44? Does it mean something else?
5. zwol commented on Nov 27, 2015

   

   zwolmannequin

   on Nov 27, 2015

   MannequinAuthor

   More actions

   FWIW, the actual behavior of \w matching "everything in Unicode general categories L* and N*, plus U+005F (underscore)" is consistent across all versions I can conveniently test (2.7, 3.4, 3.5).

   In 2.7, there are four characters in general category Nl that \w doesn't match, but I believe that is just a bug, not an intentional difference of behavior.
6. ezio-melotti

   added

   topic-regex

   type-featureA feature request or enhancementA feature request or enhancement

   on Jan 4, 2016
7. iritkatriel commented on Dec 1, 2021

   

   iritkatriel

   on Dec 1, 2021

   Member

   More actions

   It's too late for the 2.7 docs, but the current docs can still be updated.
8. iritkatriel

   added

   easy

   3.9 (EOL)end of lifeend of life

   3.10 (EOL)end of lifeend of life

   3.11only security fixesonly security fixes

   on Dec 1, 2021
9. iritkatriel

   changed the title [-]Clarify exactly what \w matches in UNICODE mode[/-] [+][doc] Clarify exactly what \w matches in UNICODE mode[/+] on Dec 1, 2021
10. slateny commented on Feb 28, 2022

    

    slateny

    on Feb 28, 2022

    Contributor

    More actions

    Would a change like this be accurate?

    Matches Unicode word characters; this includes most alphanumeric characters as well as the underscore. In Unicode, alphanumeric characters are defined to be the general categories L + N (see https://unicode.org/reports/tr44/#General_Category_Values). If the :const:`ASCII` flag is used, only ``[a-zA-Z0-9_]`` is matched.
11. ezio-melotti

    transferred this issue fromon Apr 10, 2022
12. JelleZijlstra

    added a commit that references this issue on Dec 20, 2022

    gh-69929: re docs: Add more specific definition of \w (#92015)

    36a0b1d
13. miss-islington

    added 4 commits that reference this issue on Dec 20, 2022

    pythongh-69929: re docs: Add more specific definition of \w (pythonGH…

    02c8cc3

    pythongh-69929: re docs: Add more specific definition of \w (pythonGH…

    ee3aeb9

    gh-69929: re docs: Add more specific definition of \w (GH-92015)

    b81d1c3

    gh-69929: re docs: Add more specific definition of \w (GH-92015)

    cfa78ec
14. slateny

    closed this as completedon Dec 20, 2022
15. jonburdo

    added a commit that references this issue on Dec 20, 2022

    pythongh-69929: re docs: Add more specific definition of \w (python#9…

    f41cef6
16. iritkatriel

    added a commit that references this issue on Dec 28, 2022

    Merged main into regmachine branch (#50)

    fdc73cb

Issue actions