GeezerVison

linakhan470-cell left a comment

File and readline tokenizers keep every byte they read because the lexer owns
buffer growth and repairs its pointers after reallocations. Long inputs
therefore grow tokenizer memory with the entire source.

Let the reader reuse a bounded input window when no token or formatted string
needs older bytes. Track the absolute offset of that window and save pointer
offsets only when backing storage moves.

Tokenizer results expose pointers into the active input buffer. That ties every
consumer to the buffer lifetime and prevents the reader from reusing older
storage.

Return logical source spans with their start and end locations instead. Pegen
and `_tokenize` materialize a short-lived view before requesting another token,
and the tokenizer no longer keeps a second end pointer for the last token.