GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — More strict rules for group numbers and names in RE · Issue #91760 · python/cpython

More strict rules for group numbers and names in RE #91760

New issue
Copy link
New issue
Copy link
Closed
#91794
#91792
Closed
More strict rules for group numbers and names in RE#91760
#91794
#91792
Copy link
Labels
3.11only security fixesonly security fixestopic-regextype-featureA feature request or enhancementA feature request or enhancement

Description

@serhiy-storchaka
serhiy-storchaka
opened on Apr 20, 2022
Issue body actions

There were unintentional changes in parsing regular expressions between Python 2 and Python 3.

  1. Group references.

    In patterns and replacement strings you can refer a group by its number using syntax \N where N is a 1-2 digit decimal number. The number should not start by 0, because it will be in an octal escape sequence. The group number can also be used in the conditional expression (?(N)...) in patterns and in references \g<N> in replacement strings. And it is interesting, that in Python 3 it can be not only a sequence of decimal digits. The following things are allowed in the group number:

    • Initial zero: \g<01>.
    • Spaces around the number: \g< 1 >.
    • Underscores: \g<1_2>.
    • Non-decimal digits: \g<¹>.
    • Non-ASCII decimal digits: \g<१>.

    All this is purely an implementation artifact. After \g< we search the nearest > and pass a substring between < and > to int(). In other implementation we could search the longest sequence of decimal digits and all above examples (except may be the first one) would be filtered out automatically.

  2. Group names.

    In (?P<name>...), (?P=name), (?(name)...) and \g<name> we can refer groups by name. To avoid ambiguity there is a limitation: the name should follow the rules for identifier. In Python 2 it means that it should contain only letters, digits and underscores and start with a non-digit. Letters and digits are ASCII-only: [A-Za-z] and [0-9].

    In Python 3 identifiers can contain non-ASCII letters and digits. It is good. But in bytes patterns and replacement strings the codes \xaa, \xb2, \xb3, \xb5, \xb9, \xba, \xc0-\xd6, \xd8-\xf6, \xf8-\xff are allowed in the group name. They correspond characters ª²³µ¹ºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ after decoding.

    It is an implementation artifact too. Bytes patterns and replacement strings are decoded with the Latin1 encoding for parsing. It simplifies and speeds up the code. There is no other reason why letters and digits in the range U-0080--U-00FF are allowed.

    Note that In Python 3 the bytes literal can only contain printable literal characters in the ASCII range. Codes outside of this range should be represented as octal or hexadecimal escape sequences. So supporting non-ASCII letters and digits does not add to readability.

Since the above "features" are not intentional, not supported by most other RE engines (except regex, which is also written in Python), are not tested, and can be changed in result of refactoring the parser, I suggest to introduce more strict rules on group number and name.

  1. Group number should only contain ASCII decimal digits in range [0-9]. Initial 0 is not allowed except for group number 0.
  2. Group name in the bytes pattern or replacement string should only contain ASCII letters and digits.

The question: do we need a deprecation period for this? I have wrote a code for both options (with deprecation and with error), will create PRs tomorrow.

Activity

  1. serhiy-storchaka
    added
    type-featureA feature request or enhancementA feature request or enhancement
    topic-regex
    3.11only security fixesonly security fixes
    on Apr 20, 2022
  2. ezio-melotti commented on Apr 23, 2022

    @ezio-melotti
    ezio-melotti
    on Apr 23, 2022
    Member
    More actions

    The question: do we need a deprecation period for this?

    A deprecation period of at least one release would be good.

  3. serhiy-storchaka commented on Apr 24, 2022

    @serhiy-storchaka
    serhiy-storchaka
    on Apr 24, 2022
    MemberAuthor
    More actions

    Could you please review the PR?

    I am not sure about forbidding the initial zero. On one hand, \012 is not a backreference to group 12, but a character with code 0o12, so there may be some confusion. On other hand, PCRE (according to online testers) allows the initial zero and interprets the number as decimal. So I perhaps will remove this check.

  4. serhiy-storchaka
    closed this as completedin #91794on Apr 30, 2022
  5. serhiy-storchaka
    added a commit that references this issue on Apr 30, 2022

    gh-91760: Deprecate group names and numbers which will be invalid in …

    19dca04
  6. serhiy-storchaka
    reopened this on Apr 30, 2022
  7. serhiy-storchaka
    closed this as completedin #91792on May 8, 2022
  8. serhiy-storchaka
    added a commit that references this issue on May 8, 2022

    gh-91760: More strict rules for numerical group references and group …

    a84a56d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.11only security fixesonly security fixestopic-regextype-featureA feature request or enhancementA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.