GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-153569: hide tokenizer state behind token and source operations by pablogsal · Pull Request #156654 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Aug 30, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.

Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.

Part of gh-153569.

Previous: #156484. Next: #157055.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot mentioned this pull request Aug 30, 2026
Base the tokenizer API on source offsets #153569
Open
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch 4 times, most recently from b2bc8d1 to d9b2b16 Compare August 30, 2026 13:16
@pablogsal pablogsal added the skip news label Aug 30, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch 4 times, most recently from 9ce1fd9 to afd9d72 Compare September 5, 2026 17:46
This was referenced Sep 5, 2026
gh-153569: return token spans and unify decoded source storage #156482
Merged
gh-153569: centralize formatted-string state and source spans #156484
Merged
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch from afd9d72 to f72cb2b Compare September 5, 2026 19:17
@pablogsal pablogsal changed the title gh-153569: simplify tokenizer source and formatted-string state gh-153569: finish tokenizer offsets, diagnostics, and opaque API Sep 5, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch 3 times, most recently from 7e5d82c to 9240806 Compare September 6, 2026 20:57
@pablogsal pablogsal changed the title gh-153569: finish tokenizer offsets, diagnostics, and opaque API gh-153569: finish tokenizer offsets and the opaque API cutover Sep 6, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch 4 times, most recently from 0eb2ce0 to 6f220e4 Compare September 6, 2026 23:35
@pablogsal pablogsal closed this Sep 6, 2026
@pablogsal pablogsal reopened this Sep 6, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch from 6f220e4 to bbc061f Compare September 7, 2026 00:03
@pablogsal pablogsal changed the title gh-153569: finish tokenizer offsets and the opaque API cutover gh-153569: hide tokenizer state behind token and source operations Sep 7, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch from bbc061f to b619385 Compare September 7, 2026 00:05
@pablogsal pablogsal mentioned this pull request Sep 7, 2026
gh-153569: consolidate tokenizer input, layout, and diagnostics #157055
Merged
pablogsal added 2 commits September 9, 2026 00:26
@pablogsal
pythongh-153569: hide tokenizer state behind token and source operations
5caac0f
@pablogsal
pythongh-153569: keep persistent scanner positions as source offsets
97b6803
pablogsal added 2 commits September 9, 2026 00:26
@pablogsal
pythongh-153569: include token types and owned metadata in tokenizer …
1d56ef5 
…results
@pablogsal
pythongh-153569: keep character scanning inline with source offsets
aef6c9e
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-source-ftstring-clean branch 2 times, most recently from 0b47836 to aef6c9e Compare September 8, 2026 23:27
@pablogsal
pablogsal marked this pull request as ready for review September 8, 2026 23:28
@pablogsal
pablogsal requested review from a team, AA-Turner, emmatyping, ericsnowcurrently, erlend-aasland, itamaro and lysnikolaou as code owners September 8, 2026 23:28
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 8, 2026
@pablogsal
pablogsal merged commit 7bfa97f into python:main Sep 12, 2026
62 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal
pablogsal deleted the gh-153569-tokenizer-source-ftstring-clean branch September 12, 2026 13:30
@bedevere-app bedevere-app Bot removed the awaiting core review label Sep 12, 2026
clin1234 pushed a commit to clin1234/cpython that referenced this pull request Sep 12, 2026
@pablogsal @clin1234
pythongh-153569: hide tokenizer state behind token and source operati…
4c60f27 
…ons (python#156654)

Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.

Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
johnslavik pushed a commit to johnslavik/cpython that referenced this pull request Sep 14, 2026
@pablogsal @johnslavik
pythongh-153569: hide tokenizer state behind token and source operati…
5b5bd38 
…ons (python#156654)

Make pegen and _tokenize consume tokenizer operations instead of accessing the tokenizer layout. Return token kind, span, location, and raw-string context together, so parser actions use the context of the cached token.

Represent persistent scanner positions as offsets and remove pointer save/restore bookkeeping when input storage grows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping is a code owner
@itamaro itamaro Awaiting requested review from itamaro itamaro is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@pablogsal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.