GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-155292: Skip updating unicodedata with mismatched interpreter by encukou · Pull Request #157066 · python/cpython

Conversation

@encukou

@encukou encukou commented Sep 7, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.

This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.

The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The #155292 fix only considered that case.

Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.

For an extra complication, download the input (rcf3454.txt) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may break URL encoding in urllib.

  • Issue: stringprep and IDNA 2003 incorrectly handles some characters #155292

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
encukou added 3 commits September 7, 2026 15:07
@encukou
pythongh-155292: Skip updating unicodedata with mismatched interpreter
f9648ba
mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.

This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.

The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The pythonGH-155292 fix only considered that case.

Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.

As an extra complication, download the input (RFC 3454) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may prevent downloads.
@encukou
Move the now-shared DATA_DIR up
3f0b50f
@encukou
Add blurb
4d5d35a
@encukou
encukou requested a review from sethmlarson September 7, 2026 13:14
@bedevere-app bedevere-app Bot mentioned this pull request Sep 7, 2026
stringprep and IDNA 2003 incorrectly handles some characters #155292
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 7, 2026
encukou
encukou commented Sep 7, 2026
View reviewed changes
Comment thread Tools/unicode/mkstringprep.py
data_file = open(FILENAME, encoding='utf-8')

with data_file:
with open("Tools/unicode/data/rfc3454.txt", encoding='utf-8') as data_file:

@encukou encukou Sep 7, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured we don't need the urllib call here: you shouldn't hit this in the usual case, and if you do, the FileNotFoundError makes it clear that you need rfc3454.txt and where to put it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead and sethmlarson reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@encukou
Remove unused import
7d844df
gpshead
gpshead approved these changes Sep 8, 2026
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 8, 2026
sethmlarson
sethmlarson approved these changes Sep 8, 2026
View reviewed changes
maurycy
maurycy reviewed Sep 8, 2026
View reviewed changes
Comment thread Tools/unicode/makeunicodedata.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@encukou @maurycy
Apply typo fix from @maurycy
e9f441b
Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
@encukou
encukou merged commit c42790b into python:main Sep 9, 2026
51 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Sep 9, 2026
@encukou
encukou deleted the fix-makeunicode-update branch September 9, 2026 13:57
maurycy added a commit to maurycy/cpython that referenced this pull request Sep 10, 2026
@maurycy
Merge branch 'main' into tachyon-native-callable
e028ff5
* main: (158 commits)
  pythongh-156924: Try reifying lazy imports in `ForwarRef.evaluate()` (python#156940)
  pythongh-156233: Fix typos and code snippet errors in the docs (python#156475)
  pythongh-156837: Refer to yield expressions in generator function definitions (pythonGH-156863)
  pythongh-155292: Skip updating unicodedata with mismatched interpreter (pythonGH-157066)
  pythongh-157170: Document the scope of global curses settings (pythonGH-157207)
  pythongh-155966: Correct handling of `math.tanpi` poles (python#155980)
  pythongh-157170: Restore use_env() after test_use_prescr_screen in test_curses (pythonGH-157171)
  pythongh-156910: fix deadlock in type_set_abstractmethods under free-threading (python#156948)
  pythongh-155648: Write the empty and placeholder IDLE tests (python#156260)
  pythongh-153569: centralize formatted-string state and source spans (python#156484)
  pythongh-157137: Mark the PEP 820 soft deprecations as 3.15, not `next` (python#157138)
  pythongh-152433: Use regular LoadLibrary in UWP for Windows system libs (pythonGH-156972)
  pythongh-121617: Fix Py_CLEAR() memcpy in C++: replace NULL with _Py_NULL (python#157188)
  pythongh-157135: Fix documentation errors in the `math.atan{2}pi` functions (python#157136)
  pythongh-121617: Fix Py_CLEAR() in C++: replace NULL with _Py_NULL (python#157067)
  pythongh-156774: Speed up pdb startup with asyncio guard (python#156775)
  pythongh-156109: Allow static, non-framework iOS builds (python#156110)
  pythongh-156780: Emscripten: add missing EM_JS_DEPS (python#156798)
  pythongh-152936: Make privileged functions available on Android (python#152977)
  pythongh-123018: Keep the libedit history file header when truncating (pythonGH-157165)
  ...
clin1234 pushed a commit to clin1234/cpython that referenced this pull request Sep 12, 2026
@encukou @maurycy @clin1234
pythongh-155292: Skip updating unicodedata with mismatched interpreter (
8927af2
pythonGH-157066)

mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.

This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.

The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The pythonGH-155292 fix only considered that case.

Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.

As an extra complication, download the input (RFC 3454) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may prevent downloads.

Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
johnslavik pushed a commit to johnslavik/cpython that referenced this pull request Sep 14, 2026
@encukou @maurycy @johnslavik
pythongh-155292: Skip updating unicodedata with mismatched interpreter (
a1d1e02
pythonGH-157066)

mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.

This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.

The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The pythonGH-155292 fix only considered that case.

Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.

As an extra complication, download the input (RFC 3454) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may prevent downloads.

Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@sethmlarson sethmlarson sethmlarson approved these changes
+1 more reviewer
@maurycy maurycy maurycy left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @maurycy @gpshead @sethmlarson
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.