GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-153568: Parse arithmetic with operator loops by pablogsal · Pull Request #157413 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 13, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

For a + b + c, the parser first finds a, then calls the rule again to get a + b, then again to get (a + b) + c. It puts each partial result in the parse cache so the next call can pick it up. That works, but it adds rule calls, cache lookups and cache updates to something that is really just a loop.

This adds an (operator_loop) flag to the six arithmetic and bitwise rules. The generated code parses a, keeps the current expression in a local variable, and loops over + b and + c. Each step builds the next AST node directly. It only puts the finished expression in the cache, so all that bookkeeping between steps goes away.

This also helps when there is no operator. Even a plain name passes through these rules, and the general parser still has to check whether it can grow the expression. The loop can just see that the next token isn't an operator and return.

The grammar and AST actions stay the same apart from the flag, and the existing error rules are still there.

The table compares parsing with and without these loops. Both builds include pending changes that reduce allocations, skip grammar choices using the next token, make stack checks and cache updates cheaper, and handle simple one-token expressions directly.

Codebase Python files Release CPU time PGO/LTO CPU time
Standard library 2,060 16.02% less 17.91% less
Home Assistant 18,731 17.19% less 19.18% less
mypy 442 16.50% less 17.83% less
PyTorch 4,890 14.54% less 17.50% less

The timings include tokenization, building the AST and cleanup, without compiling bytecode or running the files.

  • Issue: Speed up the parser #153568

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt reacted with eyes emoji
All reactions
  • 👀 1 reaction
@pablogsal
pablogsal requested review from AA-Turner and lysnikolaou as code owners September 13, 2026 10:47
@bedevere-app bedevere-app Bot mentioned this pull request Sep 13, 2026
Speed up the parser #153568
Open
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 13, 2026
@pablogsal
pablogsal force-pushed the gh-153568-parser-operator-loops branch from 1c7ff67 to b98d899 Compare September 14, 2026 20:08
pablogsal added 2 commits September 14, 2026 21:36
@pablogsal
pythongh-153568: Parse arithmetic with operator loops
5d8e0ef
Turn rules marked with operator_loop into a simple loop over operators and
operands. Keep the cached result and the existing error rules.
@pablogsal
pythongh-153568: Fix generator typing and the news path
500def1
@pablogsal
pablogsal force-pushed the gh-153568-parser-operator-loops branch from b98d899 to 500def1 Compare September 14, 2026 20:50
@waroad waroad mentioned this pull request Oct 17, 2026
importlib: later import hangs forever after a RecursionError in a small-stack thread leaves a module lock held #159652
Closed
eendebakpt
eendebakpt reviewed Oct 18, 2026
View reviewed changes
Comment thread Tools/peg_generator/pegen/c_generator.py
candidates: dict[str, Rhs] = {}
for name, rule in marked.items():
if rule.type != "expr_ty":
reject(name, "requires result type expr_ty")

@eendebakpt eendebakpt Oct 18, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not tested (and several other reject branches are not tested either). Maybe add some tests for invalid grammar?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/peg_generator/pegen/c_generator.py
left, operator, right = alt.items
assert isinstance(operator.item, StringLeaf)
token = self.exact_tokens[ast.literal_eval(operator.item.value)]
self.print(f"case {token}: {{")

@eendebakpt eendebakpt Oct 18, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No debug traces are printed (e.g. D(fprintf(...)).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ambv ambv mentioned this pull request Oct 20, 2026
gh-153568: Skip the expression precedence chain for single-token atoms #153575
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eendebakpt eendebakpt eendebakpt left review comments
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@pablogsal @eendebakpt
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.