GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-145921: Add "_DuringGC" functions for tp_traverse by encukou · Pull Request #145925 · python/cpython

Conversation

@encukou

@encukou encukou commented Mar 13, 2026 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There are newly documented restrictions on tp_traverse:

The traversal function must not have any side effects.
It must not modify the reference counts of any Python
objects nor create or destroy any Python objects.

  • Add several functions that are guaranteed side-effect-free, with a _DuringGC suffix.
  • Use these in ctypes
  • Consolidate tp_traverse docs in gcsupport.rst, moving unique content from typeobj.rst there
  • Issue: Add _DuringGC functions for tp_traverse #145921

📚 Documentation preview 📚: https://cpython-previews--145925.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@lysnikolaou @encukou
pythongh-145921: Add "_DuringGC" functions for tp_traverse
797312f
There are newly documented restrictions on tp_traverse:

    The traversal function must not have any side effects.
    It must not modify the reference counts of any Python
    objects nor create or destroy any Python objects.

* Add several functions that are guaranteed side-effect-free,
  with a _DuringGC suffix.
* Use these in ctypes
* Consolidate tp_traverse docs in gcsupport.rst, moving unique
  content from typeobj.rst there
@encukou
encukou requested review from a team, ZeroIntensity, ericsnowcurrently and markshannon as code owners March 13, 2026 15:48
@bedevere-app bedevere-app Bot added the awaiting core review label Mar 13, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Mar 13, 2026
Add _DuringGC functions for tp_traverse #145921
Closed
@encukou
Exclude PyObject_GetItemData_DuringGC from limited ABI
01cd8f7
zooba
zooba reviewed Mar 16, 2026
View reviewed changes

@zooba zooba left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly doc suggestions, the actual change LGTM!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with heart emoji
All reactions
  • ❤️ 1 reaction
Comment thread Doc/c-api/gcsupport.rst
Comment on lines +300 to +302
The traversal function has a limitation:

.. warning::

@zooba zooba Mar 16, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this kind of structure flows well in the docs? It probably works better to go straight to the warning, then in the paragraph after it replace "This means" with "The limitation on side effects means ..."

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Mar 17, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'll push back here.
The red box looks like it stands on its own. Maybe it's my banner blindness that kicks in, but my brain doesn't immediately connect "The limitation on side effects" to the box above it, so I added the line to make it a part of the running text more obviously.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@zooba zooba Mar 17, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're pushing in the same direction - I don't think the "traversal function has a limitation" flows into the warning banner (I don't think anything really flows into or out of these banners, apart from headings)

Maybe the following advice should just be rolled into the banner itself? Or maybe give the detailed advice, and then a brief warning banner afterwards for anyone who skipped over it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Mar 19, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did something similar in https://docs.python.org/3/c-api/call.html#vectorcall, but ended up also having an introductory sentence.

Could you give a concrete suggestion?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst
.. note::

The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
thread.

@zooba zooba Mar 16, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention something about "though all other Python threads are guaranteed to be blocked at the time" here?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Mar 17, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK that's an implementation detail some people want to get rid of.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@zooba zooba Mar 18, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't these new functions explicitly not thread safe, though? If we can't guarantee that you can safely access your objects during tp_traverse, what are we expecting - authors should figure out their own way to lock all other Python threads? How are they meant to coordinate it?

Maybe there's a way to describe it that says "the entire object graph is guaranteed to not be changed by any other Python thread" that leaves open the possibility of those threads not being blocked, but also not being able to interfere?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@encukou encukou Mar 19, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question -- one we should have an answer to. Are you OK leaving it to future PRs? Alas, it wouldn't be the only unclear detail around C API in free-threading.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/gcsupport.rst Outdated
Comment on lines +394 to +395
Note that these functions may fail (return ``NULL`` or -1).
Only creating and setting the exception is suppressed.

@zooba zooba Mar 16, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they fail during tp_traverse, should we be bailing out as quickly as possible? What value should we return?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Mar 17, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question; honestly I'm not sure about the intentions behind tp_traverse's result.

@pablogsal: would it be correct to say tp_traverse should return zero on recoverable errors, including being unable to visit some objects?
Currently gc.c ignores errors*, but non-zero a result makes VISIT return early; gc_free_threading.c calls PyErr_NoMemory if traversal fails.


* except _PyGC_GetReferrers (gc.get_referrers) which instead assumes that tp_traverse sets an exception on failure -- that's wrong, right?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_testmultiphase.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou added 3 commits March 17, 2026 11:16
@encukou
Fix typo in link
a94159a
@encukou
Move common info to a common section.
f5a0b07
@encukou
return m
6730065
@stestagg

stestagg commented Mar 18, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
Previous invalid comment

EDIT: Ok, please ignore this for now, the errors are almost certainly stack overflows from the fuzz input, but I'm not sure why I'm seeing it only for this PR, it's looking more likely to be an issue at my end

Ok, this is going to be a horrible comment, sorry!

I'm running some independent fuzzing on PRs, and this pr is triggering some segfaults where others don't seem to be.

Unfortunately it seems to be some sort of heisencorruption because if I run the same input in isolation then I can't reproduce, but it seems to be only happening with this PR.

Over the course of 5 hours of full-tilt fuzzing, I have 227 cores.

I'm continuing to investigate, and will update when I know more, but I wanted to put this out there slightly early incase it helps.

I think what happend was a test input for a different pr got picked up and run for this pr which exposed the bug the other pr was trying to fix. Sorry, please ignore this.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Mar 18, 2026
View reviewed changes
Comment thread Doc/c-api/gcsupport.rst

The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
if the object is immutable.

@vstinner vstinner Mar 18, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to clarify "if the object is immutable". A tuple is immutable and can contain reference cycle:

>>> t=(1,2,[])
>>> t[-1].append(t)
>>> t
(1, 2, [(...)])

Maybe something like:

"or NULL if the object doesn't contain other objects or only contains built-in scalar objects (bool, int, float, complex, bytes, str, NoneType)."

It's not easy to define when it's not needed to implement tp_clear().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Mar 19, 2026

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's for tp_clear -- I only moved those docs; I prefer to keep it out of scope of this PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/c-api/gcsupport.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_testcapi/heaptype.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou and others added 4 commits March 19, 2026 11:12
@encukou
Merge in the main branch
1eb82d7
@encukou @vstinner
Apply suggestions from code review
30a602f
Co-authored-by: Victor Stinner <[email protected]>
@encukou
Merge in the GitHub branch
2f74db1
@encukou
Some more thread safety notes
467ec09
@encukou
encukou marked this pull request as draft March 19, 2026 10:53
@bedevere-app bedevere-app Bot removed the awaiting core review label Mar 19, 2026
@encukou
Correct function name
9d1386a
vstinner
vstinner approved these changes Mar 19, 2026
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

You should copy the NEWS entry to What's New in Python 3.15.

I tested ./python -m test test_capi -R 3:3: it does not leak.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting merge label Mar 19, 2026
encukou added 2 commits March 19, 2026 13:24
@encukou
PyObject_GetItemData: parametrize raising
1cb20e6
@encukou
What's New entry
bd11e85
@encukou
encukou marked this pull request as ready for review March 19, 2026 12:27
@encukou
encukou requested a review from AA-Turner as a code owner March 19, 2026 12:27
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting merge labels Mar 19, 2026
@encukou
frogotten git add :/
df3bbfe
@encukou
Merge branch 'main' into during-gc
ba544d1
@encukou encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 25, 2026
@bedevere-bot

bedevere-bot commented Mar 25, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @encukou for commit ba544d1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F145925%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 25, 2026
@encukou
Merge branch 'main' into during-gc
3b40506
@encukou

encukou commented Apr 7, 2026

Copy link
Copy Markdown
Member Author

!buildbot s690x

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Apr 7, 2026

Copy link
Copy Markdown

The regex 's690x' did not match any buildbot builder. Is the requested builder in the list of stable builders?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented Apr 7, 2026

Copy link
Copy Markdown
Member Author

!buildbot s390x

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Apr 7, 2026

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @encukou for commit 3b40506 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F145925%2Fmerge

The command will test the builders whose names match following regular expression: s390x

The builders matched are:

  • s390x Fedora Rawhide Clang PR
  • s390x Fedora Stable Clang Installed PR
  • s390x Fedora Rawhide PR
  • s390x Fedora Stable LTO PR
  • s390x Fedora Rawhide NoGIL refleaks PR
  • s390x RHEL8 Refleaks PR
  • s390x Fedora Rawhide LTO + PGO PR
  • s390x RHEL8 LTO + PGO PR
  • s390x RHEL8 LTO PR
  • s390x RHEL9 Refleaks PR
  • s390x Fedora Rawhide LTO PR
  • s390x Fedora Rawhide NoGIL PR
  • s390x Fedora Rawhide Refleaks PR
  • s390x RHEL9 LTO + PGO PR
  • s390x Fedora Stable LTO + PGO PR
  • s390x Fedora Stable PR
  • s390x Fedora Stable Clang PR
  • s390x RHEL9 PR
  • s390x Fedora Rawhide Clang Installed PR
  • s390x Fedora Stable Refleaks PR
  • s390x RHEL9 LTO PR
  • s390x RHEL8 PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou merged commit 8923ca4 into python:main Apr 8, 2026
76 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting core review label Apr 8, 2026
@encukou
encukou deleted the during-gc branch April 8, 2026 14:01
@kumaraditya303

kumaraditya303 commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

@encukou This change caused a performance regression in free-threading. See #148907

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ljfp pushed a commit to ljfp/cpython that referenced this pull request Apr 25, 2026
@encukou @lysnikolaou
@vstinner @ljfp
pythongh-145921: Add "_DuringGC" functions for tp_traverse (pythonGH-…
8061518
…145925)

There are newly documented restrictions on tp_traverse:

    The traversal function must not have any side effects.
    It must not modify the reference counts of any Python
    objects nor create or destroy any Python objects.

* Add several functions that are guaranteed side-effect-free,
  with a _DuringGC suffix.
* Use these in ctypes
* Consolidate tp_traverse docs in gcsupport.rst, moving unique
  content from typeobj.rst there

Co-authored-by: Lysandros Nikolaou <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
@vstinner

vstinner commented Apr 27, 2026

Copy link
Copy Markdown
Member

@encukou This change caused a performance regression in free-threading. See #148907

Issue fixed by 29917d5.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zooba zooba zooba left review comments
@vstinner vstinner vstinner approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity ZeroIntensity is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@encukou @stestagg @bedevere-bot @kumaraditya303 @vstinner @zooba @lysnikolaou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.