GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Free-threading support: StopTheWorld synchronization + per-object locking · Issue #52 · zhuyifei1999/guppy3

Free-threading support: StopTheWorld synchronization + per-object locking #52

New issue
Copy link
New issue
Copy link
Closed
Closed
Free-threading support: StopTheWorld synchronization + per-object locking#52
Copy link

Description

@devdanzin
devdanzin
opened on Apr 4, 2026
Issue body actions

Context

guppy3's heapy module walks the interpreter's heap, thread states, and frame stacks — operations that read data owned by other threads. Under free-threaded Python (PEP 703), these operations need explicit synchronization. The sets module needs per-object critical sections.

This issue tracks the full set of changes needed to make guppy3 safe under free-threaded Python. A detailed synchronization guide is available at https://gist.github.com/devdanzin/c430e676ac2c27b30af644847af9aa1b (produced by ft-review-toolkit agents).

Key insight: the STW contract on 3.14+

On Python 3.14+ free-threading builds, object allocation does NOT trigger GC — GC runs only on the eval breaker in the bytecode loop, which doesn't execute during StopTheWorld. This means PyList_New, PyDict_New, PyUnicode_FromString, Py_BuildValue, etc. are all safe during STW. The only hard constraint is: no Python code invocation (__hash__, __eq__, __getattr__, __str__, __del__). Dict operations are safe if keys/values are CheckExact-verified builtins.

This significantly simplifies the implementation — worklists can use PyList, results can be built during STW, and the two-phase "collect C data during STW / build Python objects after" pattern is only needed for operations that genuinely invoke Python code.

Tasks

1. rootstate.c — Interpreter/thread state walking [CRITICAL]

Functions: rootstate_relate_unlocked, rootstate_traverse_unlocked, rootstate_getattr_unlocked, rootstate_dir_unlocked (lines 314–895)

These walk the interpreter and thread state linked lists (interps, threads, frames, dicts). Currently protected only by HEAD_LOCK, which guards list membership but not the contents being read. Without the GIL, another thread could be pushing/popping frames, modifying ts->current_exception, or changing ts->dict concurrently.

Fix: Wrap each function in _PyEval_StopTheWorld(interp) / _PyEval_StartTheWorld(interp). On 3.14+, the allocation calls these functions make (PyUnicode_FromFormat, PyList_Append, Py_VISIT) are all safe during STW, so no two-phase split is needed — the entire function body can run with the world stopped.

2. hv.c, hv_cli_dictof.c — Full heap traversal [CRITICAL]

Functions: hv_heap / NyHeapView_iterate (hv.c:857–964), hv_cli_dictof_update (hv_cli_dictof.c:80–138)

Heap traversal follows references recursively using tp_traverse. Without STW, objects can be deallocated or mutated during traversal. hv_cli_dictof_update calls gc.get_objects(), then iterates the result.

Fix: _PyEval_StopTheWorld for the traversal phase. Since allocation is safe during STW on 3.14+, the PyList-based worklist can remain as-is (no C-level stack replacement needed). NyNodeGraph_AddEdge and PyList_Append are safe. gc.get_objects() invokes Python so it must be called before STW.

For dict-based memoization during STW, use CheckExact guards on keys/values to avoid __hash__/__eq__ callbacks — this approach was confirmed correct.

3. horizon.c — Linked list + PyRefTracer race [HIGH]

Lines: 33–85, 171–260

The Horizon linked list is already well-protected with PyMutex rm_mutex. Two remaining issues:

  • Install/uninstall race (lines 62–85): The PyRefTracer_GetTracer check is racy (noted in a code comment). Fix: keep rm_mutex held during horizon_install/horizon_uninstall.
  • rm.replaced flag (line 55): Written from the tracer callback, read in horizon_news. Fix: use _Py_atomic_store_int / _Py_atomic_load_int.

4. bitset.c — _Py_ForgetReference realloc optimization [HIGH]

Lines: 892–978

The realloc-in-place optimization uses _Py_ForgetReference and _Py_DecRefTotal — internal CPython APIs whose behavior under free-threading is not guaranteed. The PyUnstable_Object_IsUniquelyReferenced guard is necessary but may not be sufficient.

Fix: Either restructure to avoid internal APIs (allocate new → copy → decref old), or wrap in Py_BEGIN_CRITICAL_SECTION(self) and verify behavior on each CPython release.

5. stdtypes.c — Frame/type traversal [HIGH, inherits STW]

Lines: 380–460, 586–666

Frame traversal reads _PyInterpreterFrame internals; type traversal reads tp_dict, tp_subclasses, tp_mro. These are called as part of heap traversal (task 2), so they inherit STW protection from the caller. No additional synchronization needed.

Note: NyStaticType_GetState accesses type->tp_subclasses as a raw integer index — this CPython internal detail may differ under free-threading and needs testing.

6. hv_cli_prod.c — sizeof_PyGC_Head lazy init [MEDIUM]

Lines: 25–78

Classic TOCTOU on a static variable. The initialization calls PyImport_ImportModule, which can release the GIL.

Fix: Double-checked locking with PyMutex + _Py_atomic_load_ssize_relaxed / _Py_atomic_store_ssize_relaxed.

7. impsets.c — nodeset_exports import [LOW, already correct]

Already uses PyMutex with double-checked locking. Minor improvement: add _Py_atomic_load_ptr_relaxed / _Py_atomic_store_ptr_relaxed for formal correctness on ARM.

8. Sets module — Per-object critical sections

Add Py_BEGIN_CRITICAL_SECTION(self) / Py_END_CRITICAL_SECTION(self) to all methods that mutate set state. This is the easiest module to make thread-safe.

9. GIL declarations — Phased migration

Both modules currently declare Py_MOD_GIL_USED (correct for now). Recommended phasing:

  1. Sets module: add critical sections → switch to Py_MOD_GIL_NOT_USED
  2. Heapyc: add STW to traversal/rootstate → switch to Py_MOD_GIL_NOT_USED when complete

Debug tooling suggestion

An assert_world_stopped() / assert_world_running() annotation system (similar to Linux kernel's might_sleep()) would catch transitive STW violations at debug time. A possible implementation using _Thread_local is described in the synchronization guide linked above.

Recommended implementation order

  1. Sets module (easiest, self-contained) — quick win, builds confidence
  2. Easy heapy fixes — sizeof_PyGC_Head mutex, horizon.c atomics, impsets.c atomic reads
  3. Debug annotations — assert_world_stopped() / assert_world_running()
  4. rootstate.c — hardest but most impactful, wrap in STW
  5. Heap traversal — wrap in STW, gc.get_objects() before STW
  6. Switch heapyc to Py_MOD_GIL_NOT_USED — only after all above is in place

This issue was drafted and created by Claude Code and reviewed by a human.

Activity

  1. zhuyifei1999 commented on Apr 5, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 5, 2026
    Owner
    More actions

    Oops I just saw the issue creation, sorry.

    This significantly simplifies the implementation — worklists can use PyList

    I'm still thinking about this. I think there's still a benefit to using worklists, where, since it only stays for the lifetime of the stop-the-world I won't need to refcount. Considering refcounting in freethreading is lots of atomics and somewhat complex logic, this could be a lot of performance savings...

    ... but I'd have to code it correctly... I don't think a dynamically-expanding array is that difficult if at all (and pretty sure it's not).

    hv_cli_dictof_update calls gc.get_objects(), then iterates the result.

    I don't know / remember why hv_cli_dictof_update does this. I need to re-read the code. Ideally I could rewrite it so that it doesn't invoke the GC. (No other classifier does this, I don't why this one is an exception)

    The PyRefTracer_GetTracer check is racy (noted in a code comment).

    The racy comment was actually referring to other threads that may call PyRefTracer_GetTracer through other means, for example, tracemalloc. I think making horizon not race with itself makes sense but I did not think about it at the time.

    Fix: use _Py_atomic_store_int / _Py_atomic_load_int.
    Minor improvement: add _Py_atomic_load_ptr_relaxed / _Py_atomic_store_ptr_relaxed for formal correctness on ARM.

    These are nice-to-haves. For the rm.replaced the worst case that can happen is getting data that is stale. The atomic API are all CPython private APIs, annoyingly (looks like there has been requests to add to public API: https://discuss.python.org/t/exposing-atomic-operations-tooling-in-the-public-c-api/39973/9 https://discuss.python.org/t/free-threading-expose-atomic-operations/101660/6). I might try to use C11 atomics and hope it works on the builders.

    I was reading the report the other day and I saw some parts where I feel like might miss some details:

    Py_INCREF / Py_DECREF / Py_XINCREF / Py_XDECREF — atomic refcount ops

    The DECREF ones are safe only if it's not the final reference. because if it is, it could call del. Even if it's a built-in type, like say, list, it could still contain user stuffs and end up calling del on them.

    Though, I don't think heapyc should have a code path to do that. It's trying to observe world state, not mutating it by deleting existing references.

    PyUnicode_FromFormat — string creation

    As long as no %R or %S that could enter custom types I think? Are unicodes safe when repr-ed? I'm not sure if it enters locale logic, in which case would be unsafe

  2. zhuyifei1999 commented on Apr 5, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 5, 2026
    Owner
    More actions
    1. hv_cli_prod.c — sizeof_PyGC_Head lazy init [MEDIUM]

    Actually I just remembered. In free-threading this should just return 0. It's not used there python/cpython@587d480

  3. zhuyifei1999
    added 3 commits that reference this issue on Apr 5, 2026

    horizon.c & impsets.c: Use atomics

    47c9909

    hv_cli_prod.c: Remove sizeof_PyGC_Head offset if Py_GIL_DISABLED

    356bb90

    heapyc: Add initial version of stop-the-world traverse & relate

    f5a8c0b
  4. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions

    Looks like guppy3 will be incompatible if Py_DEBUG && Py_GIL_DISABLED, for 3.14.3.

    Trying to debug a refcount issue with Py_DEBUG, then I hit this backtrace:

    #0  0x00007ffff7d5081c in ?? () from /usr/lib64/libc.so.6
    #1  0x00007ffff7cf6926 in raise () from /usr/lib64/libc.so.6
    #2  0x00007ffff7cde36a in abort () from /usr/lib64/libc.so.6
    #3  0x00007ffff7cdf580 in ?? () from /usr/lib64/libc.so.6
    #4  0x00007ffff7ceedef in __assert_fail () from /usr/lib64/libc.so.6
    #5  0x00005555558793d9 in _PyCriticalSection_AssertHeld (mutex=0x7fffb4b9f55a) at cpython/Include/internal/pycore_critical_section.h:226
    #6  _Py_dict_lookup (mp=0x7fffb4b9f550, key=0x7fffb653b530, hash=0xa207ef21fd526664, value_addr=0x7ffffffee658) at cpython/Objects/dictobject.c:1254
    #7  0x000055555587c1f9 in _PyDict_GetItemRef_KnownHash_LockHeld (op=0x7fffb4b9f550, key=0x7fffb653b530, hash=0xa207ef21fd526664, result=0x7ffffffee728) at cpython/Objects/dictobject.c:2420
    #8  0x0000555555d1dc79 in bounded_lru_cache_get_lock_held (self=0x7fffb4ddc950, args=0x7fffb653b530, kwds=0x0, result=<optimized out>, key=<optimized out>, hash=<optimized out>) at cpython/Modules/_functoolsmodule.c:1325
    #9  bounded_lru_cache_wrapper (self=0x7fffb4ddc950, args=0x7fffb653b530, kwds=0x0) at cpython/Modules/_functoolsmodule.c:1499
    #10 0x0000555555d1c4d6 in lru_cache_call (op=0x7fffb4ddc950, args=0x7fffb653b530, kwds=0x0) at cpython/Modules/_functoolsmodule.c:1652
    #11 0x00005555557d6542 in _PyObject_Call (tstate=0x55555604aa80 <_PyRuntime+362816>, callable=0x7fffb4ddc950, args=0x7fffb653b530, kwargs=0x0) at cpython/Objects/call.c:361
    #12 0x00005555557d66cc in PyObject_Call (callable=0x7fffb4ddc950, args=0x7fffb653b530, kwargs=0x0) at cpython/Objects/call.c:373
    #13 0x0000555555a7e9bc in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at cpython/Python/generated_cases.c.h:2656
    

    Basically, _functools.lru_cache believes it's the sole referrer of its dict and proceeds to... not lock the dict.

    And since it's pretty trivial to get a dict to participate in GC, and I can gc.get_objects... here's a trivial reproducer for Python 3.14.3 (I stole some code from https://docs.python.org/3/library/functools.html#functools.lru_cache):

    zhuyifei1999@zhuyifei1999-Framework-16 ~/cpython $ cat /tmp/test.py 
    from functools import lru_cache
    import gc
    
    @lru_cache(maxsize=32)
    def get_pep(num):
        return type('', (), {})()
    
    for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
        pep = get_pep(n)
        print(n, pep)
    
    objs = gc.get_objects()
    
    for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
        pep = get_pep(n)
        print(n, pep)
    zhuyifei1999@zhuyifei1999-Framework-16 ~/cpython $ gdb -ex r --args cpython-tsan/bin/python3.14td /tmp/test.py
    [...]
    Using host libthread_db library "/usr/lib64/libthread_db.so.1".
    8 <__main__. object at 0x7fffb4552720>
    290 <__main__. object at 0x7fffb4552ea0>
    308 <__main__. object at 0x7fffb4553020>
    320 <__main__. object at 0x7fffb45531a0>
    8 <__main__. object at 0x7fffb4552720>
    218 <__main__. object at 0x7fffb4553320>
    320 <__main__. object at 0x7fffb45531a0>
    279 <__main__. object at 0x7fffb45534a0>
    289 <__main__. object at 0x7fffb4553620>
    320 <__main__. object at 0x7fffb45531a0>
    9991 <__main__. object at 0x7fffb45537a0>
    python3.14td: ./Include/internal/pycore_critical_section.h:226: void _PyCriticalSection_AssertHeld(PyMutex *): Assertion `cs != NULL && cs->_cs_mutex == mutex' failed.
    
    Program received signal SIGABRT, Aborted.
    0x00007ffff7d5081c in ?? () from /usr/lib64/libc.so.6
    (gdb) bt
    #0  0x00007ffff7d5081c in ?? () from /usr/lib64/libc.so.6
    #1  0x00007ffff7cf6926 in raise () from /usr/lib64/libc.so.6
    #2  0x00007ffff7cde36a in abort () from /usr/lib64/libc.so.6
    #3  0x00007ffff7cdf580 in ?? () from /usr/lib64/libc.so.6
    #4  0x00007ffff7ceedef in __assert_fail () from /usr/lib64/libc.so.6
    #5  0x00005555558793d9 in _PyCriticalSection_AssertHeld (mutex=0x7fffb466c67a) at /home/zhuyifei1999/cpython/Include/internal/pycore_critical_section.h:226
    #6  _Py_dict_lookup (mp=0x7fffb466c670, key=0x555555ff9928 <_PyRuntime+30696>, hash=8, value_addr=0x7fffffffb8f8) at /home/zhuyifei1999/cpython/Objects/dictobject.c:1254
    #7  0x000055555587c1f9 in _PyDict_GetItemRef_KnownHash_LockHeld (op=0x7fffb466c670, key=0x555555ff9928 <_PyRuntime+30696>, hash=8, result=0x7fffffffb9c8) at /home/zhuyifei1999/cpython/Objects/dictobject.c:2420
    #8  0x0000555555d1dc79 in bounded_lru_cache_get_lock_held (self=0x7fffb4be6ad0, args=0x7fffb475adb0, kwds=0x0, result=<optimized out>, key=<optimized out>, hash=<optimized out>) at /home/zhuyifei1999/cpython/Modules/_functoolsmodule.c:1325
    #9  bounded_lru_cache_wrapper (self=0x7fffb4be6ad0, args=0x7fffb475adb0, kwds=0x0) at /home/zhuyifei1999/cpython/Modules/_functoolsmodule.c:1499
    #10 0x0000555555d1c4d6 in lru_cache_call (op=0x7fffb4be6ad0, args=0x7fffb475adb0, kwds=0x0) at /home/zhuyifei1999/cpython/Modules/_functoolsmodule.c:1652
    #11 0x00005555557d4e28 in _PyObject_MakeTpCall (tstate=0x55555604aa80 <_PyRuntime+362816>, callable=0x7fffb4be6ad0, args=0x7fffffffc9b8, nargs=1, keywords=0x0) at /home/zhuyifei1999/cpython/Objects/call.c:242
    #12 0x00005555557d471c in _PyObject_VectorcallTstate (tstate=0x55555604aa80 <_PyRuntime+362816>, callable=callable@entry=0x7fffb4be6ad0, args=args@entry=0x7fffffffc9b8, nargsf=nargsf@entry=9223372036854775809, kwnames=kwnames@entry=0x0)
        at /home/zhuyifei1999/cpython/Include/internal/pycore_call.h:175
    #13 0x00005555557d6335 in PyObject_Vectorcall (callable=0x7fffb4be6ad0, args=0x7fffffffc9b8, nargsf=9223372036854775809, kwnames=0x0) at /home/zhuyifei1999/cpython/Objects/call.c:327
    #14 0x0000555555a78f50 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /home/zhuyifei1999/cpython/Python/generated_cases.c.h:1621
    #15 0x0000555555a6f0ef in _PyEval_EvalFrame (tstate=0x55555604aa80 <_PyRuntime+362816>, frame=0x7ffff6d04020, throwflag=0) at /home/zhuyifei1999/cpython/Include/internal/pycore_ceval.h:120
    #16 _PyEval_Vector (tstate=0x55555604aa80 <_PyRuntime+362816>, func=0x7fffb44b3c70, locals=0x7fffb466cbb0, args=0x0, argcount=0, kwnames=0x0) at /home/zhuyifei1999/cpython/Python/ceval.c:2110
    #17 0x0000555555a6e910 in PyEval_EvalCode (co=0x7fffb43f4a10, globals=0x7fffb466cbb0, locals=0x7fffb466cbb0) at /home/zhuyifei1999/cpython/Python/ceval.c:982
    #18 0x0000555555bd6e05 in run_eval_code_obj (tstate=tstate@entry=0x55555604aa80 <_PyRuntime+362816>, co=co@entry=0x7fffb43f4a10, globals=globals@entry=0x7fffb466cbb0, locals=locals@entry=0x7fffb466cbb0)
        at /home/zhuyifei1999/cpython/Python/pythonrun.c:1366
    #19 0x0000555555bd69b9 in run_mod (mod=mod@entry=0x7fffb44f1640, filename=filename@entry=0x7fffb4788c10, globals=globals@entry=0x7fffb466cbb0, locals=locals@entry=0x7fffb466cbb0, flags=0x7fffffffcf48, arena=arena@entry=0x7fffb4050970, 
        interactive_src=0x0, generate_new_source=0) at /home/zhuyifei1999/cpython/Python/pythonrun.c:1460
    #20 0x0000555555bd24f2 in pyrun_file (fp=fp@entry=0x725000000000, filename=filename@entry=0x7fffb4788c10, start=start@entry=257, globals=globals@entry=0x7fffb466cbb0, locals=locals@entry=0x7fffb466cbb0, closeit=closeit@entry=1, 
        flags=0x7fffffffcf48) at /home/zhuyifei1999/cpython/Python/pythonrun.c:1294
    #21 0x0000555555bd0aff in _PyRun_SimpleFileObject (fp=0x725000000000, filename=0x7fffb4788c10, closeit=1, flags=0x7fffffffcf48) at /home/zhuyifei1999/cpython/Python/pythonrun.c:521
    #22 0x0000555555bd013b in _PyRun_AnyFileObject (fp=0x725000000000, filename=0x7fffb4788c10, closeit=1, flags=0x7fffffffcf48) at /home/zhuyifei1999/cpython/Python/pythonrun.c:81
    #23 0x0000555555c28d2d in pymain_run_file_obj (program_name=0x7fffb44d38f0, filename=0x7fffb4788c10, skip_source_first_line=<optimized out>) at /home/zhuyifei1999/cpython/Modules/main.c:410
    #24 pymain_run_file (config=config@entry=0x555556014340 <_PyRuntime+139776>) at /home/zhuyifei1999/cpython/Modules/main.c:429
    #25 0x0000555555c27b31 in pymain_run_python (exitcode=0x7fffffffd04c) at /home/zhuyifei1999/cpython/Modules/main.c:695
    #26 Py_RunMain () at /home/zhuyifei1999/cpython/Modules/main.c:776
    #27 0x0000555555c28268 in pymain_main (args=args@entry=0x7fffffffd350) at /home/zhuyifei1999/cpython/Modules/main.c:806
    #28 0x0000555555c28346 in Py_BytesMain (argc=2, argv=0x7fffffffd4d8) at /home/zhuyifei1999/cpython/Modules/main.c:830
    #29 0x00005555556c67a5 in main (argc=2, argv=0x7fffffffd4d8) at /home/zhuyifei1999/cpython/Programs/python.c:15
    #30 0x00007ffff7ce047e in ?? () from /usr/lib64/libc.so.6
    #31 0x00007ffff7ce059b in __libc_start_main () from /usr/lib64/libc.so.6
    #32 0x000055555560ace5 in _start ()
    

    @devdanzin Is this worth reporting?

    That said, I don't think this is a dealbreaker for guppy since we will stop-the-world for all traversal, so while we look at objects, no one else will.

  5. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions

    Another common code path that is almost always unsafe for STW:

    • PyErr_CheckSignals may call _Py_RunGC (if pending GC flag was set on thread)
    • PyErr_SetFromErrno may call PyErr_CheckSignals (if EINTR)

    I don't think it's super relevant to guppy, but it might help spot issues in other cexts in case they use stop-the-world. It means almost anything that interfaces with the OS (which may call PyErr_SetFromErrno) / long running procedures (which may call PyErr_CheckSignals) are illegal for STW...

    Though for guppy it's kinda a bummer because I can't interrupt a large heap traversal, unless I find a different way to do it:

    guppy3/src/heapy/hv.c

    Lines 313 to 318 in 859957d

    static int
    xt_hd_traverse(struct ExtraType *xt, PyObject *obj, visitproc visit, void *arg)
    {
    PyErr_CheckSignals();
    if (PyErr_Occurred())
    return -1;

  6. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions

    Performance is struggling. -DNDEBUG builds in both cases:

    from guppy import hpy, heapy
    hpy().test(False)
    heapy.heapyc.xmemstats()

    Non-freethreading: 13.0s, Freethreading: 17.6s

    Long traversal test:

    class node(object):
        def __init__(self, value, _next=None):
            self.value = value
            self.next = _next
    
    llist = node(0)
    curnode = llist
    for i in range(int(1e6)):
        curnode.next = node(0)
        curnode = curnode.next
    
    hp = hpy()
    print(hp.heap())
    print(hp.iso(curnode).rp)

    Non-freethreading: 3.4s, Freethreading: 15.6s

    It's hard to tell how much time was spent on refcounting (since it's split time and inlined a lot, so won't show up on flame graphs), but I think there's merit to this:

    I think there's still a benefit to using worklists, where, since it only stays for the lifetime of the stop-the-world I won't need to refcount.

    Especially considering how I can't check for signals anymore.

  7. devdanzin commented on Apr 6, 2026

    @devdanzin
    devdanzin
    on Apr 6, 2026
    Author
    More actions

    from functools import lru_cache
    import gc

    @lru_cache(maxsize=32)
    def get_pep(num):
    return type('', (), {})()

    for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
    pep = get_pep(n)
    print(n, pep)

    objs = gc.get_objects()

    for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
    pep = get_pep(n)
    print(n, pep)

    Is this worth reporting?

    I was told anything involving gc.get_objects() is likely to result in FT issues (python/cpython#132707 (comment)), but if we can find another way of getting that dictionary to participate in GC it's sure an important bug.

    I've given Claude the mission of finding another way, will report back when it's finished.

  8. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions

    we can find another way of getting that dictionary to participate in GC it's sure an important bug.

    I mean, you could just call a lru_cache-ed function with GC-able arguments and that dict will also participate in GC. I think the main issue is more of obtaining a reference to that dict without gc.get_objects() (or guppy I guess), and for _functools.lru_cache, I don't think it'd willingly give its cache out anywhere.

    I was told anything involving gc.get_objects() is likely to result in FT issues (python/cpython#132707 (comment))

    I got it repro-ed in rr. This is a tuple being printed, which is obvious from your backtrace, but with rr I can see why it is a bad value.

    This is the backtrace of when the tuple was being printed:

    [#0] 0x7fc55c7a406b → PyUnicodeWriter_WriteRepr(writer=0x313820907c0, obj=0x0)
    [#1] 0x7fc55c751ded → tuple_repr(self=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
    [#2] 0x7fc55c723f65 → PyObject_Repr(v=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
    [#3] 0x7fc55c7a4083 → PyUnicodeWriter_WriteRepr(writer=0x31382090a00, obj=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
    [#4] 0x7fc55c6eb2e0 → list_repr_impl(v=0x31382041340)
    [#5] 0x7fc55c6eb2e0 → list_repr(self=[<ModuleSpec(_cached='/usr/lib/python3.14t/__pycache__/_ast_unparse.cpython-314.pyc', _set_fileattr=True, _uninitialized_submodules=[], loader=<SourceFileLoader(name='_ast_unparse', path='/usr/lib/python3.14t/_ast_unparse.py') at remote 0x31376750130>, loader_state=None, name='_ast_unparse', origin='/usr/lib/python3.14t/_ast_unparse.py', submodule_search_locations=None, _initializing=False) at remote 0x313768000b0>, <_Precedence at remote 0x31376780590>, <_Precedence at remote 0x31376780550>, <_Precedence at remote 0x31376780510>, <_Precedence at remote 0x313767804d0>, <_Precedence at remote 0x31376780490>, <_Precedence at remote 0x31376780450>, <_Precedence at remote 0x31376780410>, <_Precedence at remote 0x313767803d0>, <_Precedence at remote 0x31376780390>, <_Precedence at remote 0x31376780310>, <_Precedence at remote 0x313767802d0>, <_Precedence at remote 0x31376780290>, <_Precedence at remote 0x31376780250>, <_Precedence at remote 0x31376780210>, <_Precedence at remote 0x313767801d0>, <_Precedence at rem...(truncated))
    [#6] 0x7fc55c723f65 → PyObject_Repr(v=[<ModuleSpec(_cached='/usr/lib/python3.14t/__pycache__/_ast_unparse.cpython-314.pyc', _set_fileattr=True, _uninitialized_submodules=[], loader=<SourceFileLoader(name='_ast_unparse', path='/usr/lib/python3.14t/_ast_unparse.py') at remote 0x31376750130>, loader_state=None, name='_ast_unparse', origin='/usr/lib/python3.14t/_ast_unparse.py', submodule_search_locations=None, _initializing=False) at remote 0x313768000b0>, <_Precedence at remote 0x31376780590>, <_Precedence at remote 0x31376780550>, <_Precedence at remote 0x31376780510>, <_Precedence at remote 0x313767804d0>, <_Precedence at remote 0x31376780490>, <_Precedence at remote 0x31376780450>, <_Precedence at remote 0x31376780410>, <_Precedence at remote 0x313767803d0>, <_Precedence at remote 0x31376780390>, <_Precedence at remote 0x31376780310>, <_Precedence at remote 0x313767802d0>, <_Precedence at remote 0x31376780290>, <_Precedence at remote 0x31376780250>, <_Precedence at remote 0x31376780210>, <_Precedence at remote 0x313767801d0>, <_Precedence at rem...(truncated))
    [#7] 0x7fc55c81db47 → _PyEval_EvalFrameDefault(tstate=0x5570742d9c30, frame=<optimized out>, throwflag=<optimized out>)
    [#8] 0x7fc55c8313de → _PyEval_EvalFrame(tstate=0x5570742d9c30, frame=<optimized out>, throwflag=0x0)
    [#9] 0x7fc55c8313de → _PyEval_Vector(tstate=0x5570742d9c30, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
    

    Same as what you saw. But with rr I can set a watchpoint that location to see where it was set in the past:

    (rr) gef➤  watch -l v->ob_item[1]
    Hardware watchpoint 1: -location v->ob_item[1]
    (rr) gef➤  reverse-continue 
    Continuing.
    [...]
    (rr) gef➤  reverse-continue 
    Continuing.
    [Switching to Thread 699164.699397]
    
    Thread 17 hit Hardware watchpoint 1: -location v->ob_item[1]
    
    Old value = 0x0
    New value = <list_iterator at remote 0x31362a70600>
    0x00007fc55c949c3f in PyTuple_SET_ITEM (op=(<str_ascii_iterator at remote 0x31362a705d0>, <list_iterator at remote 0x31362a70600>), index=0x1, value=0x0) at /usr/src/debug/dev-lang/python-0.3.14.3/Python-3.14.3/Include/cpython/tupleobject.h:37
    37	    tuple->ob_item[index] = value;
    
    (rr) gef➤  py-bt
    Traceback (most recent call first):
      File "python3.14t/traceback.py", line 672, in output_line
        caret_group = list(group)
      File "python3.14t/traceback.py", line ?, in format_frame_summary
        (failed to get frame line number)
      File "python3.14t/traceback.py", line 764, in format
        formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
      File "python3.14t/traceback.py", line 1510, in format
        yield from _ctx.emit(exc.stack.format(colorize=colorize))
      File "python3.14t/traceback.py", line 1574, in print
        for line in self.format(chain=chain, colorize=colorize):
      File "python3.14t/traceback.py", line 134, in print_exception
        te.print(file=file, chain=chain, colorize=colorize)
      File "python3.14t/traceback.py", line 143, in _print_exception_bltin
        return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
      File "python3.14t/threading.py", line 1333, in invoke_excepthook
        hook(args)
      File "python3.14t/threading.py", line 1084, in _bootstrap_inner
        self._invoke_excepthook(self)
      File "python3.14t/threading.py", line 1044, in _bootstrap
        self._bootstrap_inner()
    (rr) gef➤  bt
    #0  0x00007fc55c949c3f in PyTuple_SET_ITEM (op=(<str_ascii_iterator at remote 0x31362a705d0>, <list_iterator at remote 0x31362a70600>), index=0x1, value=0x0)
        at Python-3.14.3/Include/cpython/tupleobject.h:37
    #1  zip_longest_next (op=<itertools.zip_longest at remote 0x313620403c0>) at Python-3.14.3/Modules/itertoolsmodule.c:3867
    #2  0x00007fc55c68ec23 in iternext (iter=<optimized out>, item=<synthetic pointer>) at Python-3.14.3/Objects/abstract.c:2870
    #3  PyIter_Next (iter=<optimized out>) at Python-3.14.3/Objects/abstract.c:2920
    #4  0x00007fc55c94893d in groupby_step (gbo=0x31362820420) at Python-3.14.3/Modules/itertoolsmodule.c:512
    #5  _grouper_next (op=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Modules/itertoolsmodule.c:666
    #6  0x00007fc55c6ec3d6 in list_extend_iter_lock_held (self=0x31362120440, iterable=<unknown at remote 0x9>) at Python-3.14.3/Objects/listobject.c:1263
    #7  _list_extend (self=self@entry=0x31362120440, iterable=iterable@entry=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Objects/listobject.c:1452
    #8  0x00007fc55c6ed0ce in list___init___impl (self=self@entry=0x31362120440, iterable=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Objects/listobject.c:3497
    #9  0x00007fc55c6ed1cb in list_vectorcall (type=<optimized out>, args=0x7fc533dfe738, nargsf=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Objects/listobject.c:3521
    #10 0x00007fc55c81f971 in _PyEval_EvalFrameDefault (tstate=0x5570742de380, frame=<optimized out>, throwflag=<optimized out>) at Python-3.14.3/Python/generated_cases.c.h:2153
    #11 0x00007fc55c8313de in _PyEval_EvalFrame (tstate=0x5570742de380, frame=<optimized out>, throwflag=0x0) at Python-3.14.3/Include/internal/pycore_ceval.h:120
    #12 _PyEval_Vector (tstate=0x5570742de380, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Python/ceval.c:2083
    #13 0x00007fc55c6adbb6 in _PyObject_VectorcallTstate (tstate=0x5570742de380, callable=<function at remote 0x31356804500>, args=0x7fc533dfe908, nargsf=0x8000000000000001, kwnames=0x0)
        at Python-3.14.3/Include/internal/pycore_call.h:169
    #14 PyObject_CallOneArg (func=func@entry=<function at remote 0x31356804500>, arg=arg@entry=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',))
        at Python-3.14.3/Objects/call.c:395
    #15 0x00007fc55c8b0c0b in _PyErr_Display (file=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>, unused=<optimized out>, 
        value=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',), tb=<optimized out>) at Python-3.14.3/Python/pythonrun.c:1157
    #16 0x00007fc55c963e20 in thread_excepthook_file (file=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>, exc_type=<type at remote 0x7fc55cb66040>, 
        exc_value=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',), exc_traceback=<traceback at remote 0x31362120340>, 
        thread=<Thread(_context=<_contextvars.Context at remote 0x313546ac740>, _daemonic=False, _ident=140484955600576, _initialized=True, _invoke_excepthook=<function at remote 0x31354c92640>, _name='Thread-228 (stress_list)', _native_id=699397, _os_thread_handle=<_thread._ThreadHandle at remote 0x31354153000>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x31354692610>, _waiters=<collections.deque at remote 0x313543dd800>, acquire=<built-in method acquire of _thread.lock object at remote 0x31354692610>, locked=<built-in method locked of _thread.lock object at remote 0x31354692610>, release=<built-in method release of _thread.lock object at remote 0x31354692610>) at remote 0x313547c42f0>, _flag=True) at remote 0x31354621510>, _stderr=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>) at remote 0x313547b3030>) at Python-3.14.3/Modules/_threadmodule.c:2242
    #17 thread_excepthook (module=<optimized out>, args=<optimized out>) at Python-3.14.3/Modules/_threadmodule.c:2321
    #18 0x00007fc55c81db47 in _PyEval_EvalFrameDefault (tstate=0x5570742de380, frame=<optimized out>, throwflag=<optimized out>) at Python-3.14.3/Python/generated_cases.c.h:2488
    #19 0x00007fc55c8313de in _PyEval_EvalFrame (tstate=0x5570742de380, frame=<optimized out>, throwflag=0x0) at Python-3.14.3/Include/internal/pycore_ceval.h:120
    #20 _PyEval_Vector (tstate=0x5570742de380, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Python/ceval.c:2083
    #21 0x00007fc55c6b1816 in _PyObject_VectorcallTstate (tstate=0x5570742de380, callable=<function at remote 0x313546ce700>, args=0x7fc533dfede8, nargsf=0x1, kwnames=0x0)
        at Python-3.14.3/Include/internal/pycore_call.h:169
    #22 method_vectorcall (method=<optimized out>, args=0x7fc55cbbba90 <_PyRuntime+115536>, nargsf=0x0, kwnames=0x0) at Python-3.14.3/Objects/classobject.c:73
    #23 0x00007fc55c964178 in thread_run (boot_raw=0x5570742d9330) at Python-3.14.3/Modules/_threadmodule.c:359
    #24 0x00007fc55c8cbd2b in pythread_wrapper (arg=<optimized out>) at Python-3.14.3/Python/thread_pthread.h:242
    #25 0x00007fc55c4cc972 in ?? () from /usr/lib64/libc.so.6
    #26 0x00007fc55c53d6e4 in clone () from /usr/lib64/libc.so.6
    

    (I trimmed the paths a bit because it was way too spammy)

    It's trying to print the traceback and for some reason itertools wants to set an item to NULL on an tuple that is participating in GC: https://github.com/python/cpython/blob/v3.14.3/Modules/itertoolsmodule.c#L3867

    List I think normally the NULL elements in a new tuple does not leave the function it's constructed in, and since gc.get_objects does a STW which synchronizes all threads, it will never see a tuple half-constructed.

    Maybe there's a reason itertools does this. I can't tell (it's not obvious to me what the code is doing without reading more closely).

  9. devdanzin commented on Apr 6, 2026

    @devdanzin
    devdanzin
    on Apr 6, 2026
    Author
    More actions

    Here's what Claude concluded when trying to avoid gc:

    The question is: what commit is YiFei's build from? If it's before commit 9ad0c7b0f14 (the gh-132641 fix from around March 2025), this is already fixed. If it's after, something else is going on.

    Could you ask YiFei:

    1. What's python3.14td --version or git log --oneline -1 on their CPython build?
    2. Does git log --oneline Modules/_functoolsmodule.c | head -5 show the gh-132641 fix?

    If the build predates the fix, there's no point trying to find a gc-free reproducer — the bug is already known and the fix is to use the non-_LockHeld dict APIs. If it's after the fix, then gc.get_objects() is triggering a regression or a different code path, and that would be worth reporting regardless of whether we can remove the gc dependency.

  10. devdanzin commented on Apr 6, 2026

    @devdanzin
    devdanzin
    on Apr 6, 2026
    Author
    More actions

    Here's a reproducer for the itertools issue, we should file this as a bug in the same class as python/cpython#146056:

    # Crash in itertools.zip_longest: ittuple contains NULL after iterator exhaustion
    # gc.get_referents() exposes the internal tuple, repr() crashes on NULL item
    #
    # On builds with the PyUnicodeWriter_WriteRepr NULL fix (3.13.x, 3.14.x+),
    # this displays <NULL> instead of crashing. On builds without it, segfault.
    # Either way, zip_longest should not expose NULLs in a GC-tracked tuple.
    
    import itertools
    import gc
    
    zl = itertools.zip_longest(iter([1]), iter([1, 2, 3]))
    next(zl)  # (1, 1) — both iterators alive
    next(zl)  # (None, 2) — first iterator exhausted, ittuple[0] set to NULL
    
    # The internal ittuple now contains NULL at position 0
    # gc.get_referents exposes it
    refs = gc.get_referents(zl)
    for r in refs:
        if isinstance(r, tuple):
            print(repr(r))  # Segfault or <NULL> depending on build
  11. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions
    zhuyifei1999@zhuyifei1999-Framework-16 ~/cpython $ git log --oneline Modules/_functoolsmodule.c | grep 132641
    66d6860439c [3.14] gh-132641: fix race in `lru_cache` under free-threading (GH-133787) (#133979)
    

    Yes

    Looking at that commit, it's more like it's related to the cause of the crash rather than fixing it. The change (python/cpython@66d6860439c#diff-d1deea0fb29260d389bac1b24c3ec3631e843d177e85c0e93d5cf50966dee630R1456) modifies the code to call the unlocked methods, while the unlocked methods assert that the lock has been held, causing the assert fail. Normally this would be find because lru_cache has exclusive ownership over the dict, but with a call to gc, that bet is off.

    Here's a reproducer for the itertools issue, we should file this as a bug in the same class as python/cpython#146056:

    Yep, this repro is much nicer. Do you want to file it? I'm not very familiar with filing CPython issues 😅

  12. devdanzin commented on Apr 6, 2026

    @devdanzin
    devdanzin
    on Apr 6, 2026
    Author
    More actions

    Do you want to file it? I'm not very familiar with filing CPython issues 😅

    Sure, probably tonight.

    Here's Claude's analysis, your first reproducer is the lru_cache bug that doesn't go away when we fix the zip_longest one:


    This is a legitimate CPython bug in the lru_cache free-threading fix. The reproducer needs gc.get_objects() because only the GC's stop-the-world mechanism creates the state where the assertion detects the lock mismatch (normal single-threaded execution without GC interaction happens to work because there's no contention, but the lock is genuinely not held).

    Summary of what we have — two independent bugs:

    1. zip_longest NULL-in-ittuple — stores NULL in a GC-tracked tuple when iterators exhaust. Our patch fixes it. Worth filing standalone.

    2. lru_cache critical section doesn't lock the cache dict — bounded_lru_cache_wrapper does Py_BEGIN_CRITICAL_SECTION(self) but _PyDict_GetItemRef_KnownHash_LockHeld asserts self->cache is locked. The gh-132641 fix converted the SetItem calls to _LockHeld variants (assuming self's CS covers the dict), but the assertion checks the dict's own mutex. Fix: change line 1498 to Py_BEGIN_CRITICAL_SECTION2(self, self->cache) (and similarly at line 1511). This is a real bug in the gh-132641 fix, reproducible only on debug free-threaded builds because release builds don't have the assertion.

    Reacted by zhuyifei1999
  13. zhuyifei1999 commented on Apr 6, 2026

    @zhuyifei1999
    zhuyifei1999
    on Apr 6, 2026
    Owner
    More actions

    change line 1498 to Py_BEGIN_CRITICAL_SECTION2(self, self->cache) (and similarly at line 1511)

    oh, I didn't think about that. Makes sense yeah.

    (normal single-threaded execution without GC interaction happens to work because there's no contention, but the lock is genuinely not held)

    Uh, I don't think this is the case. The assertion is bypassed if the object has a refcount of 1:

    https://github.com/python/cpython/blob/a95ee3a21d97afdbe6bd2ce4cd8343a36cd13b02/Include/internal/pycore_critical_section.h#L64-L67

    lru_cache assumes it's the exclusive owner of that dict, and that assumption is not held when you have gc.get_objects(). Stop-the-world is not relevant here.

  14. zhuyifei1999
    added 3 commits that reference this issue on Apr 6, 2026

    Remove two cases of recursive locking

    49c8b4a

    heapyc: Assert dict accesses are builtin immutables

    eda4a2c

    hv.c: Use array-based worklist with no refcounting

    7319360
  15. zhuyifei1999 commented on Oct 23, 2026

    @zhuyifei1999
    zhuyifei1999
    on Oct 23, 2026
    Owner
    More actions

    Freethreading branch merged into master

  16. zhuyifei1999
    closed this as completedon Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402639 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.