Description

pablogsal

opened on Jul 11, 2026

Issue body actions

The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings.

I think tokenizer positions should be offsets into the decoded source instead:

typedef Py_ssize_t TokenizerOffset;

typedef struct {
    TokenizerOffset start;
    TokenizerOffset end;
} TokenizerSpan;

The main ideas would be:

• One source object owns the decoded text.
• The cursor only stores its current offset and line boundaries.
• Tokens, errors and f-string state use offset spans instead of pointers.
• pegen and _tokenize ask the source for a view or copy of a span.
• Sequential tokenization keeps the current line in the cursor, while uncommon line lookups can use a small sparse index.
• Normal parsing can keep one contiguous buffer, while tokenize(readline) can eventually use reclaimable chunks.

For incremental tokenization, new decoded input would be appended only when the cursor needs more data. Existing offsets would remain valid even if the underlying storage moves. Once tokenize has returned copied token and line strings, old input could be discarded when no active token, cursor, f-string frame or error still refers to it.

This is very nice because it separates source storage from tokenizer state, removes pointer rebasing, and means consumers no longer need to access tokenizer internals.

Linked PRs

• gh-153569: Split tokenizer lexer into focused files #153585
• gh-153569: Add tokenizer source and cursor primitives #153587
• gh-153569: Replace tokenizer input readers #156472
• gh-153569: return token spans and unify decoded source storage #156482
• gh-153569: centralize formatted-string state and source spans #156484
• gh-153569: hide tokenizer state behind token and source operations #156654
• gh-153569: consolidate tokenizer input, layout, and diagnostics #157055
• gh-153569: Simplify tokenizer state #158124
• gh-153569: Simplify tokenizer source views and reader ownership #158849

Activity

1. Arbaaz123676 commented on Jul 11, 2026

   

   Arbaaz123676

   on Jul 11, 2026

   More actions

   Hi @pablogsal, I'd like to work on this issue. Can you please assign it to me?
2. pablogsal commented on Jul 11, 2026

   

   pablogsal

   on Jul 11, 2026

   MemberAuthor

   More actions

   Hi @pablogsal, I'd like to work on this issue. Can you please assign it to me?

   Thanks for the help @Arbaaz123676! Unfortunately this is something I am working on right now as a series of commits and is very delicate work. If you want to help reviewing the PRs that would be good!

   Reacted by Arbaaz Ahmed
3. pablogsal

   added 3 commits that reference this issue on Jul 11, 2026

   gh-153569: Split tokenizer lexer into focused files (#153585)

   a2d3787

   pythongh-153569: Add tokenizer source and cursor primitives

   2682fe5

   pythongh-153569: Add tokenizer source and cursor primitives

   1bf43aa
4. picnixz

   added

   type-featureA feature request or enhancementA feature request or enhancement

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Jul 12, 2026
5. pablogsal

   added 3 commits that reference this issue on Jul 18, 2026

   fixup! pythongh-153569: Add tokenizer source and cursor primitives

   edf6d0a

   fixup! pythongh-153569: Add tokenizer source and cursor primitives

   b19b995

   fixup! pythongh-153569: Add tokenizer source and cursor primitives

   0dea29d
6. pablogsal

   added 5 commits that reference this issue on Aug 25, 2026

   pythongh-153569: Add tokenizer source and cursor primitives

   697e7dd

   fixup! pythongh-153569: Add tokenizer source and cursor primitives

   100072c

   fixup! pythongh-153569: Add tokenizer source and cursor primitives

   4b0454f

   gh-153569: Add tokenizer source and cursor primitives (#153587)

   f54fd2a

   pythongh-153569: preserve unterminated interactive input

   adf62b2
7. 225 remaining items

   Load more actions
8. pablogsal

   added 10 commits that reference this issue on Oct 5, 2026

   pythongh-153569: Use logical spans in tokenizer views and column calc…

   3f78561

   pythongh-153569: Move tokenizer input state and encoding ownership in…

   723a678

   pythongh-153569: Keep tokenizer source scans and copies on offsets an…

   f36c44b

   pythongh-153569: Move tokenizer source access into the source API

   8f7e822

   pythongh-153569: Use logical spans in tokenizer views and column calc…

   2b46f42

   pythongh-153569: Move tokenizer input state and encoding ownership in…

   1cfbc93

   pythongh-153569: Keep tokenizer source scans and copies on offsets an…

   e60d602

   pythongh-153569: Remove redundant tokenizer views and simplify reader…

   7da4c7f

   pythongh-153569: Remove redundant tokenizer views and simplify reader…

   ffb781d

   pythongh-153569: Reuse the incremental decoder factory helper

   33d219c
9. pablogsal

   added 2 commits that reference this issue on Oct 13, 2026

   pythongh-153569: Export incremental decoder helper for parser generat…

   d386e50

   gh-153569: Simplify tokenizer source views and reader ownership (#158849

   f892b11
10. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026
11. github-project-automation

    moved this to Todo in Project_VKon Oct 18, 2026
12. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026

Issue actions