GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-135676: Simplify docs on lexing names by encukou · Pull Request #140464 · python/cpython

Conversation

encukou commented Oct 22, 2025 •

edited by github-actions Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
normalizes the name
validates the name, using the id_start/id_continue sets (referred to in previous sections as “letter-like” and “number-like” characters, with a link to the details)

This also means we don't need xid_start/xid_continue to define the behaviour :)

Issue: Reword the Lexical Analysis chapter of the docs #135676

📚 Documentation preview 📚: https://cpython-previews--140464.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

encukou and others added 4 commits

October 8, 2025 17:58


          Simplify Names section

Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: Micha Albert <[email protected]>
Co-authored-by: KeithTheEE <[email protected]>


          Casing; 3 dots for character ranges

6163c24


          Clean-ups

de6d1af


          Mention Unicode's *ID_Start* and *ID_Continue*

152e7aa

encukou requested review from AA-Turner and willingc as code owners

October 22, 2025 15:53

bedevere-app Bot added docs

Documentation in the Doc dir

skip news labels

Oct 22, 2025

github-project-automation Bot added this to Docs PRs

Oct 22, 2025

github-project-automation Bot moved this to Todo in Docs PRs

Oct 22, 2025

bedevere-app Bot mentioned this pull request

Oct 22, 2025

Reword the Lexical Analysis chapter of the docs #135676

Open

bedevere-app Bot added the awaiting core review label

Oct 22, 2025

StanFromIreland linked an issue

Oct 22, 2025

that may be closed by this pull request

Docs: note requirement to normalise unicode identifiers passed to globals() and locals() #86846

Closed

willingc approved these changes

Oct 22, 2025

View reviewed changes

willingc left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outstanding document @encukou. I had one small suggestion to be a bit more explicit on the normalization example with number.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Doc/reference/lexical_analysis.rst Outdated

		This means that, for example, some typographic variants of characters are
		converted to their "basic" form, for example::

		>>> nᵘₘᵇₑʳ = 3

willingc Oct 22, 2025

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to add an explicit comment that the normalized form of nᵘₘᵇₑʳis number.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji

All reactions

👍 1 reaction

encukou Oct 29, 2025

Copy link

Copy Markdown

Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this look good?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

willingc reacted with thumbs up emoji

All reactions

👍 1 reaction

bedevere-app Bot added awaiting merge and removed awaiting core review labels

Oct 22, 2025


          Make it clear that nᵘₘᵇₑʳ normalizes to number

fce5e98

encukou mentioned this pull request

Nov 4, 2025

gh-129117: Expose _PyUnicode_IsXidContinue/Start in unicodedata #140269

Merged

encukou marked this pull request as draft

November 5, 2025 10:45

bedevere-app Bot removed the awaiting merge label

Nov 5, 2025

encukou commented Nov 5, 2025

Copy link

Copy Markdown

Member Author

There was an insightful conversation in #140269. I'll update this PR to make things even clearer.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

willingc approved these changes

Nov 10, 2025

View reviewed changes

willingc left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @encukou

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-app Bot added the awaiting merge label

Nov 10, 2025

encukou added 3 commits

November 12, 2025 18:06

WIP

b9fdcf0


          Merge in the main branch

2e7f7c0


          Reword to use XID_Start and XID_Continue

43f6091

encukou marked this pull request as ready for review

November 19, 2025 16:08

bedevere-app Bot added awaiting core review and removed awaiting merge labels

Nov 19, 2025

willingc approved these changes

Nov 19, 2025

View reviewed changes

willingc left a comment

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @encukou!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-app Bot added awaiting merge and removed awaiting core review labels

Nov 19, 2025

encukou commented Nov 20, 2025

Copy link

Copy Markdown

Member Author

Thank you for the review!

@malemburg, do you also want to take a look?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou merged commit 2ff8608 into python:main

Nov 26, 2025

36 checks passed

Uh oh!

There was an error while loading. Please reload this page.

encukou deleted the lex-analysis-names-simpler branch

November 26, 2025 15:10

bedevere-app Bot removed the awaiting merge label

Nov 26, 2025

github-project-automation Bot moved this from Todo to Done in Docs PRs

Nov 26, 2025

encukou added the needs backport to 3.14 bugs and security fixes label

Nov 26, 2025

miss-islington-app Bot commented Nov 26, 2025

Copy link

Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington-app Bot commented Nov 26, 2025

Copy link

Copy Markdown

Sorry, @encukou, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 2ff8608b4da33f667960e5099a1a442197acaea4 3.14

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington-app Bot assigned encukou

Nov 26, 2025

bedevere-app Bot commented Nov 27, 2025

Copy link

Copy Markdown

GH-142015 is a backport of this pull request to the 3.14 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label

Nov 27, 2025

StanFromIreland added a commit to StanFromIreland/cpython that referenced this pull request

Nov 27, 2025


          [3.14] pythongh-135676: Simplify docs on lexing names (pythonGH-140464)

69312e4

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets

(cherry picked from commit 2ff8608)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: Micha Albert <[email protected]>
Co-authored-by: KeithTheEE <[email protected]>

encukou added a commit that referenced this pull request

Dec 3, 2025


          [3.14] gh-135676: Simplify docs on lexing names (GH-140464) (GH-142015)

79245a4

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets

(cherry picked from commit 2ff8608)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: Micha Albert <[email protected]>
Co-authored-by: KeithTheEE <[email protected]>

StanFromIreland added a commit to StanFromIreland/cpython that referenced this pull request

Dec 6, 2025


          pythongh-135676: Simplify docs on lexing names (pythonGH-140464)

9a421ea

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets


Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: Micha Albert <[email protected]>
Co-authored-by: KeithTheEE <[email protected]>

Reviewers

willingc

willingc approved these changes

AA-Turner

Awaiting requested review from AA-Turner

AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

encukou

Labels

docs

Documentation in the Doc dir

skip news

Projects

Docs PRs

Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Docs: note requirement to normalise unicode identifiers passed to globals() and locals()

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

encukou commented Oct 22, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willingc left a comment

Choose a reason for hiding this comment

Uh oh!

willingc Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

encukou Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

encukou commented Nov 5, 2025

Uh oh!

willingc left a comment

Choose a reason for hiding this comment

Uh oh!

willingc left a comment

Choose a reason for hiding this comment

Uh oh!

encukou commented Nov 20, 2025

Uh oh!

Uh oh!

miss-islington-app Bot commented Nov 26, 2025

Uh oh!

miss-islington-app Bot commented Nov 26, 2025

Uh oh!

bedevere-app Bot commented Nov 27, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

encukou commented Oct 22, 2025 •

edited by github-actions Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.