GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-95555: Support Unicode property escapes \p{...} in regular expressions by serhiy-storchaka · Pull Request #151969 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Jun 23, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Add support for \p{property} and \P{property} escapes in Unicode (str) regular expressions, for the properties the engine can resolve without the unicodedata database. They are matched either as CATEGORY opcodes (character predicates and combinations of them) or as fixed sets of character ranges, so neither the matcher nor the compiler gains a unicodedata dependency.

Supported in this change:

  • many General_Category values — the groups L, N, Z, C and the values Lu, Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co and Cn;
  • the binary properties Alphabetic, Lowercase, Uppercase, Numeric, Printable, XID_Start, XID_Continue, Cased and Case_Ignorable;
  • the POSIX compatibility classes alpha, alnum, blank, cntrl, digit, graph, lower, print, space, upper, word and xdigit;
  • the code-point classes ASCII, Any, Assigned, Noncharacter_Code_Point, Join_Control and the immutable Pattern_Syntax and Pattern_White_Space.

Property and value names use loose matching (UAX #44 UAX44-LM3), and a property may be spelled \p{Lu}, \p{gc=Lu} or \p{name=yes}.

The remaining table-based properties (the General_Category values Ll/Lo and the M/P/S families, Block, and the other enumerated properties) require the unicodedata tables and are intentionally left out of this first change, to be added separately.

  • Issue: re should support \p{...} character properties #95555

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

BoboTiG reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@serhiy-storchaka @claude
pythongh-95555: Support Unicode property escapes \p{...} in regular e…
8ca0ebe
…xpressions

Add support for \p{property} and \P{property} in Unicode (str) regular
expressions, for the properties the engine can resolve without the
unicodedata database.  They are matched either as CATEGORY opcodes
(character predicates and combinations of them, see sre.c) or as fixed
sets of character ranges.

Supported properties:

* many General_Category values -- the groups L, N, Z, C and the values Lu,
  Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co and Cn;
* the binary properties Alphabetic, Lowercase, Uppercase, Numeric,
  Printable, XID_Start, XID_Continue, Cased and Case_Ignorable;
* the POSIX compatibility classes alpha, alnum, blank, cntrl, digit, graph,
  lower, print, space, upper, word and xdigit;
* the code-point classes ASCII, Any, Assigned, Noncharacter_Code_Point,
  Join_Control and the immutable Pattern_Syntax and Pattern_White_Space.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@serhiy-storchaka
serhiy-storchaka requested a review from AA-Turner as a code owner June 23, 2026 07:32
@bedevere-app bedevere-app Bot mentioned this pull request Jun 23, 2026
re should support \p{...} character properties #95555
Open
@bedevere-app bedevere-app Bot added the awaiting core review label Jun 23, 2026
@read-the-docs-community

read-the-docs-community Bot commented Jun 23, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #33315153 | 📁 Comparing 55a2db7 against main (868d9a8)

  🔍 Preview build  

23 files changed · ± 23 modified

± Modified

  • c-api/extension-modules.html
  • c-api/import.html
  • deprecations/index.html
  • deprecations/pending-removal-in-3.17.html
  • howto/abi3t-migration.html
  • library/csv.html
  • library/curses.html
  • library/curses.panel.html
  • library/dialog.html
  • library/dis.html
  • and 13 more...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

StanFromIreland
StanFromIreland reviewed Jun 23, 2026
View reviewed changes
Comment thread Doc/library/re.rst
Matches ``[0-9]`` if the :py:const:`~re.ASCII` flag is used.

__ https://www.unicode.org/versions/Unicode15.0.0/ch04.pdf#G134153
__ https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G124142

@StanFromIreland StanFromIreland Jun 23, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be added to:

cpython/Tools/unicode/makeunicodedata.py

Lines 42 to 48 in 868d9a8

# The Unicode Database
# --------------------
# When changing UCD version please update
# * Doc/library/stdtypes.rst, and
# * Doc/library/unicodedata.rst
# * Doc/reference/lexical_analysis.rst (three occurrences)
UNIDATA_VERSION = "17.0.0"

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 23, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unrelated?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@StanFromIreland StanFromIreland Jun 23, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is, if you're updating the link in this PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 24, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you mean to add this file to that list?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
StanFromIreland
StanFromIreland reviewed Jun 23, 2026
View reviewed changes
Comment thread Lib/re/_properties.py
@@ -0,0 +1,267 @@
#
# Secret Labs' Regular Expression Engine

@StanFromIreland StanFromIreland Jun 23, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't written by the company, nor is it licensed to them?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 23, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think (but I am not sure), the Secret Labs credit is an internal joke. I can drop it if nobody can confirm this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@StanFromIreland StanFromIreland Jun 23, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a joke? It was a real company, founded by Fredrik Lundh. See his bio here, for reference.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jun 23, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems I was wrong, it was a real Swedish company of Fredrik Lundh (a.k.a. "the effbot"). References to it are everywhere in the code, even in the module name _sre. So I'll leave it. We're extending their engine, not re-attributing it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka and others added 6 commits June 23, 2026 12:23
@serhiy-storchaka @claude
pythongh-95555: Add Regional_Indicator, Hex_Digit and ASCII_Hex_Digit…
388a3b6
… properties

They are complete fixed sets, matched as fixed ranges: Regional_Indicator
(the 26 symbols A..Z), ASCII_Hex_Digit (the ASCII hex digits, = POSIX
xdigit) and Hex_Digit (which adds the fullwidth forms).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@serhiy-storchaka
Update Tools/unicode/makeunicodedata.py.
f9f3931
@serhiy-storchaka @claude
Merge branch 'main' into re-properties
b32bd8f
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@serhiy-storchaka
Merge branch 'main' into re-properties
b7ffde6
# Conflicts:
#	Doc/whatsnew/3.16.rst
#	Lib/re/_parser.py
@serhiy-storchaka
Merge branch 'main' into re-properties
2d66931
@serhiy-storchaka @claude
pythongh-95555: Change the re MAGIC number to 20260622
55a2db7
Use the date the new opcodes were finalized rather than a future-dated
placeholder.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@serhiy-storchaka
serhiy-storchaka merged commit 794b42f into python:main Jun 26, 2026
55 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting core review label Jun 26, 2026
@serhiy-storchaka
serhiy-storchaka deleted the re-properties branch June 26, 2026 04:33
@serhiy-storchaka serhiy-storchaka mentioned this pull request Jun 26, 2026
gh-95555: Allow a negated property as a character set member #152245
Merged
This was referenced Jul 19, 2026
New sequences for Unicode groups and block ranges needed #43715
Closed
Recommend closing resolved or non-actionable issues (oldest-first batch, 2004–2007) #150566
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@StanFromIreland StanFromIreland StanFromIreland left review comments
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@serhiy-storchaka @StanFromIreland
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.