GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 by sethmlarson · Pull Request #155293 · python/cpython

Conversation

@sethmlarson

@sethmlarson sethmlarson commented Aug 6, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

This PR fixes makeunicodedata.py and re-runs mkstringprep.py with the new Unicode database. In the future mkstringprep.py should be run as a part of makeunicodedata.py.

I'll take on backporting this PR manually, as we'll need to apply the patch and then re-execute makeunicodedata.py for each Python version as each contains a different Unicode version.

  • Issue: stringprep and IDNA 2003 incorrectly handles some characters #155292

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@sethmlarson @StanFromIreland
Don't consider Unicode codepoint attributes outside RFC 3454
6aeb041 
Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

Co-authored-by: Stan Ulbrych <[email protected]>
@sethmlarson
sethmlarson requested review from StanFromIreland and malemburg August 6, 2026 16:53
@sethmlarson sethmlarson added type-security A security issue topic-unicode labels Aug 6, 2026
@bedevere-app bedevere-app Bot added the awaiting core review label Aug 6, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Aug 6, 2026
stringprep and IDNA 2003 incorrectly handles some characters #155292
Closed
sethmlarson added 4 commits August 6, 2026 12:12
@sethmlarson
Avoid trailing newline in stringprep.py
accf6db
@sethmlarson
encodings.idna
02a21af
@sethmlarson
Merge branch 'main' into unicode-stringprep
5ae2c33
@sethmlarson
Merge branch 'main' into unicode-stringprep
bb1bff8
@sethmlarson

sethmlarson commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@StanFromIreland @malemburg Now that GHA service has recovered, requesting a review on this PR.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
encukou
encukou reviewed Aug 11, 2026
View reviewed changes
Comment thread Lib/test/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@sethmlarson @encukou
Apply suggestion from @encukou
d978d13 
Co-authored-by: Petr Viktorin <[email protected]>
@sethmlarson
sethmlarson requested a review from encukou August 11, 2026 17:04
@encukou encukou mentioned this pull request Aug 12, 2026
Have mkstringprep.py download the RFC if it's missing sethmlarson/cpython#2
Merged
@encukou

encukou commented Aug 12, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

I filed a suggestion as a PR: sethmlarson#2. Otherwise, LGTM -- I didn't find another issue.

@malemburg would be the best person to review this, of course. Marc-André, should we wait for you?
StanFromIreland reacted with thumbs up emoji sethmlarson reacted with heart emoji
All reactions
  • 👍 1 reaction
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@encukou @sethmlarson
Have mkstringprep.py download the RFC if it's missing
35a0beb
malemburg
malemburg reviewed Aug 13, 2026
View reviewed changes

@malemburg malemburg left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have time for a review, but from casual inspection, this looks good to me. Thanks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@encukou
encukou merged commit 7e109d0 into python:main Aug 18, 2026
54 of 55 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting core review label Aug 18, 2026
@encukou

encukou commented Aug 18, 2026

Copy link
Copy Markdown
Member

Thanks for the fix!

I'll take on backporting this PR manually

OK; let me know if you want to delegate.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@sethmlarson
sethmlarson deleted the unicode-stringprep branch August 18, 2026 12:58
@encukou encukou added the needs backport to 3.15 bugs and security fixes label Aug 18, 2026
@miss-islington-app

miss-islington-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

Thanks @sethmlarson for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app

bedevere-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

GH-156017 is a backport of this pull request to the 3.15 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Aug 18, 2026
encukou added a commit to encukou/cpython that referenced this pull request Aug 18, 2026
@sethmlarson @StanFromIreland @encukou
pythongh-155292: Don't consider Unicode codepoint attributes outside …
e2ad986 
…RFC 3454 (pythonGH-155293)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
(cherry picked from commit 7e109d0)
@bedevere-app

bedevere-app Bot commented Aug 18, 2026

Copy link
Copy Markdown

GH-156020 is a backport of this pull request to the 3.14 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal pushed a commit that referenced this pull request Aug 18, 2026
@miss-islington @sethmlarson
@StanFromIreland @encukou
[3.15] gh-155292: Don't consider Unicode codepoint attributes outside…
5181304 
… RFC 3454 (GH-155293) (#156017)

gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
@stdmnpkg stdmnpkg mentioned this pull request Aug 19, 2026
python-3: update to 3.14.7 AOSC-Dev/aosc-os-abbs#17207
Merged
6 of 8 tasks
encukou added a commit that referenced this pull request Sep 4, 2026
@encukou @sethmlarson @StanFromIreland
[3.14] gh-155292: Don't consider Unicode codepoint attributes outside…
1e54caa 
… RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
Yhg1s pushed a commit that referenced this pull request Sep 8, 2026
@encukou
[3.12] gh-155292: Don't consider Unicode codepoint attributes outside…
c016c25 
… RFC 3454 (GH-155293) (GH-156020) (#156930)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)
Also, add Unicode_3_2_0_FunctionsTest as in the later versions,
to make the new test work.

Co-authored-by: Seth Larson [email protected]
Co-authored-by: Stan Ulbrych [email protected]
Co-authored-by: Petr Viktorin [email protected]
encukou added a commit that referenced this pull request Sep 9, 2026
@miss-islington @encukou
@sethmlarson @StanFromIreland
[3.13] gh-155292: Don't consider Unicode codepoint attributes outside…
c28b121 
… RFC 3454 (GH-155293) (GH-156020) (GH-156922)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>

---------

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
pablogsal pushed a commit that referenced this pull request Oct 1, 2026
@encukou @sethmlarson @StanFromIreland
[3.10] gh-155292: Don't consider Unicode codepoint attributes outside…
d397a49 
… RFC 3454 (GH-155293) (GH-156020) (#156932)

* [3.10] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>

* Add Unicode_3_2_0_FunctionsTest

* Regenerate Unicode data

---------

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
pablogsal pushed a commit that referenced this pull request Oct 1, 2026
@encukou @sethmlarson @StanFromIreland
[3.11] gh-155292: Don't consider Unicode codepoint attributes outside…
69f92eb 
… RFC 3454 (GH-155293) (GH-156020) (#156931)

* [3.11] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
(cherry picked from commit 1e54caa)

* Add Unicode_3_2_0_FunctionsTest

* Regenerate Unicode data

---------

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
icanhasmath pushed a commit to ActiveState/cpython that referenced this pull request Oct 28, 2026
@encukou @sethmlarson
@StanFromIreland @icanhasmath
[3.10] pythongh-155292: Don't consider Unicode codepoint attributes o…
eb1b45f 
…utside RFC 3454 (pythonGH-155293) (pythonGH-156020) (python#156932)

* [3.10] pythongh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (pythonGH-155293) (pythonGH-156020)

Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.

(cherry picked from commit 7e109d0)
The cherry-pick needed reworking as pythonGH-144815 wasn't backported to 3.14
and below, so unassigned characters don't have bidi values.
(cherry picked from commit 1e54caa)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>

* Add Unicode_3_2_0_FunctionsTest

* Regenerate Unicode data

---------

Co-authored-by: Seth Larson <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
(cherry picked from commit d397a49)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malemburg malemburg malemburg left review comments
@StanFromIreland StanFromIreland Awaiting requested review from StanFromIreland
@encukou encukou Awaiting requested review from encukou

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-unicode type-security A security issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@sethmlarson @encukou @malemburg
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.