GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-153569: Simplify tokenizer state by pablogsal · Pull Request #158124 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 24, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Base the tokenizer API on source offsets #153569

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@pablogsal
pablogsal requested a review from lysnikolaou as a code owner September 24, 2026 17:13
@bedevere-app bedevere-app Bot mentioned this pull request Sep 24, 2026
Base the tokenizer API on source offsets #153569
Open
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 24, 2026
@pablogsal pablogsal changed the title gh-153569: Simplify tokenizer state, input handling, and diagnostics gh-153569: Simplify tokenizer state Sep 24, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-cleanups branch 5 times, most recently from cb1af47 to 30f8f52 Compare September 24, 2026 18:08
@pablogsal
pablogsal requested a review from JelleZijlstra as a code owner September 24, 2026 19:04
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Sep 28, 2026
pablogsal added 19 commits September 29, 2026 01:00
@pablogsal
Use diagnostic line for missing-brace errors
597c294
@pablogsal
Keep implicit prepared newline metadata in the reader
9df1aa0
@pablogsal
Reuse parsed formatted string kind in lexer
578b0c5
@pablogsal
Transfer owned tokenizer encoding strings
072439a
@pablogsal
Remove unused tokenizer state and redundant branches
29f8d4a
@pablogsal
Avoid rescanning partial decoded tokenizer lines
c911eb4
@pablogsal
Reuse prepared tokenizer line terminators for CR detection
f0149a8
@pablogsal
Reuse interactive tokenizer chunks without carriage returns
c5465b3
@pablogsal
Release tokenizer constructor resources on allocation failure
02764ba
@pablogsal
Narrow interactive newline normalization wrapper
c04c0bb
@pablogsal
Close duplicated tokenizer descriptors when fdopen fails
b44bde7
@pablogsal
Preserve cached tokenizer lines when newline allocation fails
1e44edc
@pablogsal
Collect cycles involving tokenizer readline callbacks
f995e1a
@pablogsal
Preserve buffered text when decoding source files
6e43c27
@pablogsal
Clear the transferred syntax error line before cleanup
a36c95c
@pablogsal
Propagate tokenizer column conversion failures
ffd9ac3
@pablogsal
Allocate parser repetition buffers on first match
6fefac8
@pablogsal
Allocate type comment storage only when needed
cc4ae2a
@pablogsal
Reuse decoded multiline text when counting token columns
fca5aac
pablogsal and others added 12 commits September 29, 2026 01:00
@pablogsal
Skip empty decoded tokenizer chunks
98ced0a
@pablogsal
Check parser array growth before allocating
b8eba43
@pablogsal
Name tokenizer bounds and derive byte lengths from their data
a29e2fd
@pablogsal
Set the tokenizer exception when file input size overflows
3587fb1
@pablogsal
pythongh-153569: Send exact line endings in the REPL test
39f6d4c
@pablogsal
Share parser repetition buffer growth
ba72635
@pablogsal
Fold Unicode input coverage into the existing REPL test
a768c1d
@pablogsal
Keep parser array growth policy private
7264275
@pablogsal
Drop obsolete bitmap stress from source discard test
e55d8b0
@pablogsal
Preserve input failures while scanning string literals
714d6dd
@pablogsal
fixup! Preserve input failures while scanning string literals
657d92c
@pablogsal
fixup! fixup! Preserve input failures while scanning string literals
39436f7
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-cleanups branch from fd32bb0 to 39436f7 Compare September 29, 2026 00:03
@pablogsal pablogsal removed the type-feature A feature request or enhancement label Sep 29, 2026
@pablogsal
pythongh-153569: Remove assertion-only lexer bookkeeping
452e6fa
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Sep 29, 2026
@pablogsal pablogsal removed the type-feature A feature request or enhancement label Oct 5, 2026
@pablogsal
pablogsal merged commit 0b8aa89 into python:main Oct 5, 2026
85 of 87 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal
pablogsal deleted the gh-153569-tokenizer-cleanups branch October 5, 2026 13:28
@bedevere-app bedevere-app Bot removed the awaiting core review label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@JelleZijlstra JelleZijlstra Awaiting requested review from JelleZijlstra JelleZijlstra is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@pablogsal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.