GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-136757: Reuse static strings for exact operator tokens by omkar-334 · Pull Request #151838 · python/cpython

Conversation

@omkar-334

@omkar-334 omkar-334 commented Jun 21, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR reduces repeated string allocations in the tokenizer by reusing static interned strings for exact multi-character operator tokens such as ==, ->, //=, and ....

Before -

"==" repeated 1,000 times -> 1,000 distinct "==" string objects

After -

"==" repeated 1,000 times -> 1 distinct "==" string object
  • Issue: Intern string representation of operators and some other symbolic literals #136757

Benchmarks

These are local synthetic measurements from pyperf --rigorous on a quieter machine. The public tokenize timing cases are the strongest timing evidence; one direct _tokenize.TokenizerIter streaming case still had a pyperf stability warning.

Workload:

  • 2,000 repetitions of a source snippet containing all 24 candidate multi-character exact operators.
  • Compared public tokenize.tokenize() and private _tokenize.TokenizerIter.
  • Measured both streaming token consumption and retained token streams.

Object reuse:

Repeated "==" tokens:
before: 1,000 distinct str objects
after:      1 distinct str object

Retained operator token strings:
before: 48,000 distinct str objects for 24 values
after:      24 distinct str objects for 24 values

Peak memory for retained token streams:

list(tokenize.tokenize):
before: 70.091 MiB
after:  68.113 MiB
change: -2.82%

list(_tokenize.TokenizerIter):
before: 65.054 MiB
after:  63.076 MiB
change: -3.04%

Timing:

tokenize.tokenize count:
51.0 ms +- 0.7 ms -> 48.5 ms +- 0.8 ms
1.05x faster

list(tokenize.tokenize):
91.1 ms +- 1.6 ms -> 88.6 ms +- 1.1 ms
1.03x faster

_tokenize.TokenizerIter count:
28.6 ms +- 0.6 ms -> 26.8 ms +- 0.7 ms
1.07x faster
note: this case had a pyperf stability warning

list(_tokenize.TokenizerIter):
51.8 ms +- 1.0 ms -> 50.7 ms +- 0.6 ms
1.02x faster

ast.parse control:
no significant difference

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@omkar-334
Reuse static strings for exact operator tokens
8be2a0c
@omkar-334
omkar-334 requested review from lysnikolaou and pablogsal as code owners June 21, 2026 11:08
@bedevere-app bedevere-app Bot added the awaiting review label Jun 21, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Jun 21, 2026
Intern string representation of operators and some other symbolic literals #136757
Open
@omkar-334

omkar-334 commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

cc @ZeroIntensity for review
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
omkar-334 added 2 commits June 21, 2026 16:40
@omkar-334
add news label
7d232f8
@omkar-334
Merge branch 'main' into pythongh-136757-intern-operator-strings
4a65219
@StanFromIreland
StanFromIreland requested a review from ZeroIntensity June 21, 2026 11:42
@StanFromIreland

StanFromIreland commented Jun 21, 2026

Copy link
Copy Markdown
Member

Thanks for benchmarking, could you please also share the script you used?

2,000 repetitions of a source snippet containing all 24 candidate multi-character exact operators.

So that means that 1.05x is the upper bound for a synthetic corpus, but not what anyone tokenizing real code would see. Can you please also benchmark a realistic corpus (e.g., a file from the stdlib)?

list(_tokenize.TokenizerIter):
51.8 ms +- 1.0 ms -> 50.7 ms +- 0.6 ms
1.02x faster

This is noise, the delta is below the combined error and the ranges overlap.

The change is small, but so is the win. I'm -0 on this change, but I won't object if someone else thinks it's worth it.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ZeroIntensity
ZeroIntensity reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/test/test_tokenize.py
Comment on lines +1917 to +1925
def test_old_not_equal_spelling_is_not_rewritten(self):
source = BytesIO(
b'from __future__ import barry_as_FLUFL\n'
b'a <> b\n'
)
tokens = list(tokenize._tokenize.TokenizerIter(
source.readline, encoding='utf-8', extra_tokens=True))
self.assertIn('<>', [tok[1] for tok in tokens])
self.assertNotIn('!=', [tok[1] for tok in tokens])

@ZeroIntensity ZeroIntensity Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes on main and isn't related to this change; please remove it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_tokenize.py
Comment on lines +1911 to +1912
tokens = list(tokenize._tokenize.TokenizerIter(
source.readline, encoding='utf-8', extra_tokens=True))

@ZeroIntensity ZeroIntensity Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of getting _tokenize this way, nor do I think this test stresses anything different from the first one. Let's keep just the first test and remove this one.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_tokenize.py
Comment on lines +1898 to +1900
matches = [tok.string for tok in tokens if tok.string == op]
self.assertEqual(len(matches), 2)
self.assertIs(matches[0], matches[1])

@ZeroIntensity ZeroIntensity Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use sys._is_interned instead of relying on is.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 90 days with no activity.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZeroIntensity ZeroIntensity ZeroIntensity left review comments
@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@omkar-334 @StanFromIreland @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.