GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-145668: Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER. by markshannon · Pull Request #147967 · python/cpython

Conversation

@markshannon

@markshannon markshannon commented Apr 1, 2026 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER.

  • Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
  • Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
  • Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
  • Add tp_iteritem internal slot to PyTypeObject to support a wider range of classes as virtual iterators.

This PR adds strs as virtual iterators, with the potential to add bytes, bytearray, frozenset, frozendict and others in the future.

  • Issue: Extend FOR_ITER specializations to cover other indexable sequences #145668

📚 Documentation preview 📚: https://cpython-previews--147967.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
markshannon added 2 commits April 1, 2026 13:58
@markshannon
Add FOR_ITER specialization for virtual iterators. Specialize GET_ITER.
3a6f48d
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
@markshannon
Document new tp_iteritem function slot.
74b5289
@markshannon
markshannon requested review from Fidget-Spinner, ZeroIntensity, ericsnowcurrently, savannahostrowski and tomasr8 as code owners April 1, 2026 13:37
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 1, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Apr 1, 2026
Extend FOR_ITER specializations to cover other indexable sequences #145668
Open
@markshannon
Simplify list_iteritem
4dd6992
@Fidget-Spinner

Fidget-Spinner commented Apr 1, 2026

Copy link
Copy Markdown
Member

Doesn't this change the ABI guarantees of PyTypeObject? Or are there none of those? In any case, feels strange that we are adding a public field that is opaque for our own use.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon markshannon added skip news labels Apr 1, 2026
@markshannon

markshannon commented Apr 1, 2026

Copy link
Copy Markdown
Member Author

tp_watchers is already documented as internal.
The header is public, so the fields have to go in a public header even if they aren't to be used.

Fidget-Spinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

markshannon added 7 commits April 2, 2026 12:40
@markshannon
Make it clearer to MSVC that variable doesn't escape
4376246
@markshannon
Return a small struct in tp_iteritem to support MSVC tailcalling
5a81e54
@markshannon
Fix editing slip
b927de4
@markshannon
Put limited RESUME at start of genexpr for free-threading. Fix up exc…
044cce2
…eption handling in genexpr
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
4ba6ecd
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
b08dce2
@markshannon
Fix test_dis
830068c
@markshannon
markshannon requested a review from iritkatriel as a code owner April 8, 2026 16:43
markshannon added 3 commits April 9, 2026 09:12
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
931084a
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
87215d5
@markshannon
Fix magic number
73b9b87
Fidget-Spinner
Fidget-Spinner reviewed Apr 10, 2026
View reviewed changes

@Fidget-Spinner Fidget-Spinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/typeobj.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Include/object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/bytecodes.c
_GUARD_ITER_VIRTUAL +
_PUSH_TAGGED_ZERO;

op(_GET_ITER_TRAD, (iterable -- iter, index_or_null)) {

@Fidget-Spinner Fidget-Spinner Apr 10, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does trad mean?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@markshannon markshannon Apr 13, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Traditional

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
markshannon added 3 commits April 13, 2026 14:57
@markshannon
Address review comments
fdef628
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
b8baca5
@markshannon
Make struct private
b3dc340
@Fidget-Spinner

Fidget-Spinner commented Apr 13, 2026

Copy link
Copy Markdown
Member

Do you have benchmark numbers for this?

I'm wondering of 3 extra opcodes is worth it. I think I might have a solution for the opcode problem.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon

markshannon commented Apr 14, 2026

Copy link
Copy Markdown
Member Author

Do you have benchmark numbers for this?

In the noise. But this enables more optimizations, so neutral is good.

I'm wondering of 3 extra opcodes is worth it.

The FOR_ITER_VIRTUAL is worth it, as it allows more common iterables to be treated as virtual iterators.

By themselves, the two new GET_ITER specializations probably wouldn't be.
They're worth it because they complement the FOR_ITER and, very soon, the SEND specializations.
For those specializations to work we need GET_ITER to handle the three distinct cases:

  • Already an iterator: do nothing
  • An iterable, supporting virtual iteration: push tagged zero
  • Other iterables, call __iter__ to get the iterator

So the specializations follow naturally.

I think I might have a solution for the opcode problem.

That's good, but not directly relevant to this PR I think.
We aren't in imminent danger of running out of opcodes, provided we followed a reasonably principled approach to specializations.

Fidget-Spinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon
Try to keep sphinx happy
1c269b6
@markshannon
markshannon requested review from AA-Turner, StanFromIreland and hugovk as code owners April 14, 2026 13:55
encukou
encukou reviewed Apr 14, 2026
View reviewed changes
Comment thread Include/object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/typeobj.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou
encukou reviewed Apr 14, 2026
View reviewed changes
Comment thread Include/object.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

markshannon added 3 commits April 14, 2026 15:32
@markshannon
Declare tp_iteritem and iteritemfunc private by adding _ prefix to names
09ecaef
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
92a3a97
@markshannon
Address code review: Remove struct tag
a55ac63
@Fidget-Spinner Fidget-Spinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 15, 2026
@bedevere-bot

bedevere-bot commented Apr 15, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @Fidget-Spinner for commit a55ac63 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F147967%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 15, 2026
@markshannon
Merge branch 'main' into specialize-iteration-with-jit
6406871
Fidget-Spinner
Fidget-Spinner approved these changes Apr 16, 2026
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Apr 16, 2026
@markshannon
markshannon merged commit 600f4db into python:main Apr 16, 2026
168 of 173 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Apr 16, 2026
ljfp pushed a commit to ljfp/cpython that referenced this pull request Apr 25, 2026
@markshannon @ljfp
pythonGH-145668: Add FOR_ITER specialization for virtual iterators. S…
19d02b2
…pecialize GET_ITER. (pythonGH-147967)

* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou encukou left review comments
@Fidget-Spinner Fidget-Spinner Fidget-Spinner approved these changes
@tomasr8 tomasr8 Awaiting requested review from tomasr8 tomasr8 is a code owner
@savannahostrowski savannahostrowski Awaiting requested review from savannahostrowski
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity
@iritkatriel iritkatriel Awaiting requested review from iritkatriel iritkatriel is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner
@hugovk hugovk Awaiting requested review from hugovk
@StanFromIreland StanFromIreland Awaiting requested review from StanFromIreland

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@markshannon @Fidget-Spinner @bedevere-bot @encukou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.