GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||
GeezerVison is showing github.com — Free-threading support: StopTheWorld synchronization + per-object locking · Issue #52 · zhuyifei1999/guppy3 Closed DescriptionContextguppy3's heapy module walks the interpreter's heap, thread states, and frame stacks — operations that read data owned by other threads. Under free-threaded Python (PEP 703), these operations need explicit synchronization. The sets module needs per-object critical sections. This issue tracks the full set of changes needed to make guppy3 safe under free-threaded Python. A detailed synchronization guide is available at https://gist.github.com/devdanzin/c430e676ac2c27b30af644847af9aa1b (produced by ft-review-toolkit agents). Key insight: the STW contract on 3.14+On Python 3.14+ free-threading builds, object allocation does NOT trigger GC — GC runs only on the eval breaker in the bytecode loop, which doesn't execute during StopTheWorld. This means This significantly simplifies the implementation — worklists can use Tasks1.
|
| static int | |
| xt_hd_traverse(struct ExtraType *xt, PyObject *obj, visitproc visit, void *arg) | |
| { | |
| PyErr_CheckSignals(); | |
| if (PyErr_Occurred()) | |
| return -1; |
Performance is struggling. -DNDEBUG builds in both cases:
from guppy import hpy, heapy hpy().test(False) heapy.heapyc.xmemstats()
Non-freethreading: 13.0s, Freethreading: 17.6s
Long traversal test:
class node(object): def __init__(self, value, _next=None): self.value = value self.next = _next llist = node(0) curnode = llist for i in range(int(1e6)): curnode.next = node(0) curnode = curnode.next hp = hpy() print(hp.heap()) print(hp.iso(curnode).rp)
Non-freethreading: 3.4s, Freethreading: 15.6s
It's hard to tell how much time was spent on refcounting (since it's split time and inlined a lot, so won't show up on flame graphs), but I think there's merit to this:
I think there's still a benefit to using worklists, where, since it only stays for the lifetime of the stop-the-world I won't need to refcount.
Especially considering how I can't check for signals anymore.
from functools import lru_cache
import gc@lru_cache(maxsize=32)
def get_pep(num):
return type('', (), {})()for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
pep = get_pep(n)
print(n, pep)objs = gc.get_objects()
for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
pep = get_pep(n)
print(n, pep)
Is this worth reporting?
I was told anything involving gc.get_objects() is likely to result in FT issues (python/cpython#132707 (comment)), but if we can find another way of getting that dictionary to participate in GC it's sure an important bug.
I've given Claude the mission of finding another way, will report back when it's finished.
we can find another way of getting that dictionary to participate in GC it's sure an important bug.
I mean, you could just call a lru_cache-ed function with GC-able arguments and that dict will also participate in GC. I think the main issue is more of obtaining a reference to that dict without gc.get_objects() (or guppy I guess), and for _functools.lru_cache, I don't think it'd willingly give its cache out anywhere.
I was told anything involving gc.get_objects() is likely to result in FT issues (python/cpython#132707 (comment))
I got it repro-ed in rr. This is a tuple being printed, which is obvious from your backtrace, but with rr I can see why it is a bad value.
This is the backtrace of when the tuple was being printed:
[#0] 0x7fc55c7a406b → PyUnicodeWriter_WriteRepr(writer=0x313820907c0, obj=0x0)
[#1] 0x7fc55c751ded → tuple_repr(self=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
[#2] 0x7fc55c723f65 → PyObject_Repr(v=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
[#3] 0x7fc55c7a4083 → PyUnicodeWriter_WriteRepr(writer=0x31382090a00, obj=(<str_ascii_iterator at remote 0x31362a705d0>, 0x0))
[#4] 0x7fc55c6eb2e0 → list_repr_impl(v=0x31382041340)
[#5] 0x7fc55c6eb2e0 → list_repr(self=[<ModuleSpec(_cached='/usr/lib/python3.14t/__pycache__/_ast_unparse.cpython-314.pyc', _set_fileattr=True, _uninitialized_submodules=[], loader=<SourceFileLoader(name='_ast_unparse', path='/usr/lib/python3.14t/_ast_unparse.py') at remote 0x31376750130>, loader_state=None, name='_ast_unparse', origin='/usr/lib/python3.14t/_ast_unparse.py', submodule_search_locations=None, _initializing=False) at remote 0x313768000b0>, <_Precedence at remote 0x31376780590>, <_Precedence at remote 0x31376780550>, <_Precedence at remote 0x31376780510>, <_Precedence at remote 0x313767804d0>, <_Precedence at remote 0x31376780490>, <_Precedence at remote 0x31376780450>, <_Precedence at remote 0x31376780410>, <_Precedence at remote 0x313767803d0>, <_Precedence at remote 0x31376780390>, <_Precedence at remote 0x31376780310>, <_Precedence at remote 0x313767802d0>, <_Precedence at remote 0x31376780290>, <_Precedence at remote 0x31376780250>, <_Precedence at remote 0x31376780210>, <_Precedence at remote 0x313767801d0>, <_Precedence at rem...(truncated))
[#6] 0x7fc55c723f65 → PyObject_Repr(v=[<ModuleSpec(_cached='/usr/lib/python3.14t/__pycache__/_ast_unparse.cpython-314.pyc', _set_fileattr=True, _uninitialized_submodules=[], loader=<SourceFileLoader(name='_ast_unparse', path='/usr/lib/python3.14t/_ast_unparse.py') at remote 0x31376750130>, loader_state=None, name='_ast_unparse', origin='/usr/lib/python3.14t/_ast_unparse.py', submodule_search_locations=None, _initializing=False) at remote 0x313768000b0>, <_Precedence at remote 0x31376780590>, <_Precedence at remote 0x31376780550>, <_Precedence at remote 0x31376780510>, <_Precedence at remote 0x313767804d0>, <_Precedence at remote 0x31376780490>, <_Precedence at remote 0x31376780450>, <_Precedence at remote 0x31376780410>, <_Precedence at remote 0x313767803d0>, <_Precedence at remote 0x31376780390>, <_Precedence at remote 0x31376780310>, <_Precedence at remote 0x313767802d0>, <_Precedence at remote 0x31376780290>, <_Precedence at remote 0x31376780250>, <_Precedence at remote 0x31376780210>, <_Precedence at remote 0x313767801d0>, <_Precedence at rem...(truncated))
[#7] 0x7fc55c81db47 → _PyEval_EvalFrameDefault(tstate=0x5570742d9c30, frame=<optimized out>, throwflag=<optimized out>)
[#8] 0x7fc55c8313de → _PyEval_EvalFrame(tstate=0x5570742d9c30, frame=<optimized out>, throwflag=0x0)
[#9] 0x7fc55c8313de → _PyEval_Vector(tstate=0x5570742d9c30, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
Same as what you saw. But with rr I can set a watchpoint that location to see where it was set in the past:
(rr) gef➤ watch -l v->ob_item[1]
Hardware watchpoint 1: -location v->ob_item[1]
(rr) gef➤ reverse-continue
Continuing.
[...]
(rr) gef➤ reverse-continue
Continuing.
[Switching to Thread 699164.699397]
Thread 17 hit Hardware watchpoint 1: -location v->ob_item[1]
Old value = 0x0
New value = <list_iterator at remote 0x31362a70600>
0x00007fc55c949c3f in PyTuple_SET_ITEM (op=(<str_ascii_iterator at remote 0x31362a705d0>, <list_iterator at remote 0x31362a70600>), index=0x1, value=0x0) at /usr/src/debug/dev-lang/python-0.3.14.3/Python-3.14.3/Include/cpython/tupleobject.h:37
37 tuple->ob_item[index] = value;
(rr) gef➤ py-bt
Traceback (most recent call first):
File "python3.14t/traceback.py", line 672, in output_line
caret_group = list(group)
File "python3.14t/traceback.py", line ?, in format_frame_summary
(failed to get frame line number)
File "python3.14t/traceback.py", line 764, in format
formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
File "python3.14t/traceback.py", line 1510, in format
yield from _ctx.emit(exc.stack.format(colorize=colorize))
File "python3.14t/traceback.py", line 1574, in print
for line in self.format(chain=chain, colorize=colorize):
File "python3.14t/traceback.py", line 134, in print_exception
te.print(file=file, chain=chain, colorize=colorize)
File "python3.14t/traceback.py", line 143, in _print_exception_bltin
return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
File "python3.14t/threading.py", line 1333, in invoke_excepthook
hook(args)
File "python3.14t/threading.py", line 1084, in _bootstrap_inner
self._invoke_excepthook(self)
File "python3.14t/threading.py", line 1044, in _bootstrap
self._bootstrap_inner()
(rr) gef➤ bt
#0 0x00007fc55c949c3f in PyTuple_SET_ITEM (op=(<str_ascii_iterator at remote 0x31362a705d0>, <list_iterator at remote 0x31362a70600>), index=0x1, value=0x0)
at Python-3.14.3/Include/cpython/tupleobject.h:37
#1 zip_longest_next (op=<itertools.zip_longest at remote 0x313620403c0>) at Python-3.14.3/Modules/itertoolsmodule.c:3867
#2 0x00007fc55c68ec23 in iternext (iter=<optimized out>, item=<synthetic pointer>) at Python-3.14.3/Objects/abstract.c:2870
#3 PyIter_Next (iter=<optimized out>) at Python-3.14.3/Objects/abstract.c:2920
#4 0x00007fc55c94893d in groupby_step (gbo=0x31362820420) at Python-3.14.3/Modules/itertoolsmodule.c:512
#5 _grouper_next (op=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Modules/itertoolsmodule.c:666
#6 0x00007fc55c6ec3d6 in list_extend_iter_lock_held (self=0x31362120440, iterable=<unknown at remote 0x9>) at Python-3.14.3/Objects/listobject.c:1263
#7 _list_extend (self=self@entry=0x31362120440, iterable=iterable@entry=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Objects/listobject.c:1452
#8 0x00007fc55c6ed0ce in list___init___impl (self=self@entry=0x31362120440, iterable=<itertools._grouper at remote 0x31362a70690>) at Python-3.14.3/Objects/listobject.c:3497
#9 0x00007fc55c6ed1cb in list_vectorcall (type=<optimized out>, args=0x7fc533dfe738, nargsf=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Objects/listobject.c:3521
#10 0x00007fc55c81f971 in _PyEval_EvalFrameDefault (tstate=0x5570742de380, frame=<optimized out>, throwflag=<optimized out>) at Python-3.14.3/Python/generated_cases.c.h:2153
#11 0x00007fc55c8313de in _PyEval_EvalFrame (tstate=0x5570742de380, frame=<optimized out>, throwflag=0x0) at Python-3.14.3/Include/internal/pycore_ceval.h:120
#12 _PyEval_Vector (tstate=0x5570742de380, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Python/ceval.c:2083
#13 0x00007fc55c6adbb6 in _PyObject_VectorcallTstate (tstate=0x5570742de380, callable=<function at remote 0x31356804500>, args=0x7fc533dfe908, nargsf=0x8000000000000001, kwnames=0x0)
at Python-3.14.3/Include/internal/pycore_call.h:169
#14 PyObject_CallOneArg (func=func@entry=<function at remote 0x31356804500>, arg=arg@entry=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',))
at Python-3.14.3/Objects/call.c:395
#15 0x00007fc55c8b0c0b in _PyErr_Display (file=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>, unused=<optimized out>,
value=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',), tb=<optimized out>) at Python-3.14.3/Python/pythonrun.c:1157
#16 0x00007fc55c963e20 in thread_excepthook_file (file=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>, exc_type=<type at remote 0x7fc55cb66040>,
exc_value=ValueError('Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit',), exc_traceback=<traceback at remote 0x31362120340>,
thread=<Thread(_context=<_contextvars.Context at remote 0x313546ac740>, _daemonic=False, _ident=140484955600576, _initialized=True, _invoke_excepthook=<function at remote 0x31354c92640>, _name='Thread-228 (stress_list)', _native_id=699397, _os_thread_handle=<_thread._ThreadHandle at remote 0x31354153000>, _started=<Event(_cond=<Condition(_lock=<_thread.lock at remote 0x31354692610>, _waiters=<collections.deque at remote 0x313543dd800>, acquire=<built-in method acquire of _thread.lock object at remote 0x31354692610>, locked=<built-in method locked of _thread.lock object at remote 0x31354692610>, release=<built-in method release of _thread.lock object at remote 0x31354692610>) at remote 0x313547c42f0>, _flag=True) at remote 0x31354621510>, _stderr=<_io.TextIOWrapper(mode='w') at remote 0x31354391dc0>) at remote 0x313547b3030>) at Python-3.14.3/Modules/_threadmodule.c:2242
#17 thread_excepthook (module=<optimized out>, args=<optimized out>) at Python-3.14.3/Modules/_threadmodule.c:2321
#18 0x00007fc55c81db47 in _PyEval_EvalFrameDefault (tstate=0x5570742de380, frame=<optimized out>, throwflag=<optimized out>) at Python-3.14.3/Python/generated_cases.c.h:2488
#19 0x00007fc55c8313de in _PyEval_EvalFrame (tstate=0x5570742de380, frame=<optimized out>, throwflag=0x0) at Python-3.14.3/Include/internal/pycore_ceval.h:120
#20 _PyEval_Vector (tstate=0x5570742de380, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python-3.14.3/Python/ceval.c:2083
#21 0x00007fc55c6b1816 in _PyObject_VectorcallTstate (tstate=0x5570742de380, callable=<function at remote 0x313546ce700>, args=0x7fc533dfede8, nargsf=0x1, kwnames=0x0)
at Python-3.14.3/Include/internal/pycore_call.h:169
#22 method_vectorcall (method=<optimized out>, args=0x7fc55cbbba90 <_PyRuntime+115536>, nargsf=0x0, kwnames=0x0) at Python-3.14.3/Objects/classobject.c:73
#23 0x00007fc55c964178 in thread_run (boot_raw=0x5570742d9330) at Python-3.14.3/Modules/_threadmodule.c:359
#24 0x00007fc55c8cbd2b in pythread_wrapper (arg=<optimized out>) at Python-3.14.3/Python/thread_pthread.h:242
#25 0x00007fc55c4cc972 in ?? () from /usr/lib64/libc.so.6
#26 0x00007fc55c53d6e4 in clone () from /usr/lib64/libc.so.6
(I trimmed the paths a bit because it was way too spammy)
It's trying to print the traceback and for some reason itertools wants to set an item to NULL on an tuple that is participating in GC: https://github.com/python/cpython/blob/v3.14.3/Modules/itertoolsmodule.c#L3867
List I think normally the NULL elements in a new tuple does not leave the function it's constructed in, and since gc.get_objects does a STW which synchronizes all threads, it will never see a tuple half-constructed.
Maybe there's a reason itertools does this. I can't tell (it's not obvious to me what the code is doing without reading more closely).
Here's what Claude concluded when trying to avoid gc:
The question is: what commit is YiFei's build from? If it's before commit
9ad0c7b0f14(the gh-132641 fix from around March 2025), this is already fixed. If it's after, something else is going on.Could you ask YiFei:
- What's
python3.14td --versionorgit log --oneline -1on their CPython build?- Does
git log --oneline Modules/_functoolsmodule.c | head -5show the gh-132641 fix?If the build predates the fix, there's no point trying to find a gc-free reproducer — the bug is already known and the fix is to use the non-
_LockHelddict APIs. If it's after the fix, thengc.get_objects()is triggering a regression or a different code path, and that would be worth reporting regardless of whether we can remove the gc dependency.
Here's a reproducer for the itertools issue, we should file this as a bug in the same class as python/cpython#146056:
# Crash in itertools.zip_longest: ittuple contains NULL after iterator exhaustion # gc.get_referents() exposes the internal tuple, repr() crashes on NULL item # # On builds with the PyUnicodeWriter_WriteRepr NULL fix (3.13.x, 3.14.x+), # this displays <NULL> instead of crashing. On builds without it, segfault. # Either way, zip_longest should not expose NULLs in a GC-tracked tuple. import itertools import gc zl = itertools.zip_longest(iter([1]), iter([1, 2, 3])) next(zl) # (1, 1) — both iterators alive next(zl) # (None, 2) — first iterator exhausted, ittuple[0] set to NULL # The internal ittuple now contains NULL at position 0 # gc.get_referents exposes it refs = gc.get_referents(zl) for r in refs: if isinstance(r, tuple): print(repr(r)) # Segfault or <NULL> depending on build
zhuyifei1999@zhuyifei1999-Framework-16 ~/cpython $ git log --oneline Modules/_functoolsmodule.c | grep 132641
66d6860439c [3.14] gh-132641: fix race in `lru_cache` under free-threading (GH-133787) (#133979)
Yes
Looking at that commit, it's more like it's related to the cause of the crash rather than fixing it. The change (python/cpython@66d6860439c#diff-d1deea0fb29260d389bac1b24c3ec3631e843d177e85c0e93d5cf50966dee630R1456) modifies the code to call the unlocked methods, while the unlocked methods assert that the lock has been held, causing the assert fail. Normally this would be find because lru_cache has exclusive ownership over the dict, but with a call to gc, that bet is off.
Here's a reproducer for the itertools issue, we should file this as a bug in the same class as python/cpython#146056:
Yep, this repro is much nicer. Do you want to file it? I'm not very familiar with filing CPython issues 😅
Do you want to file it? I'm not very familiar with filing CPython issues 😅
Sure, probably tonight.
Here's Claude's analysis, your first reproducer is the lru_cache bug that doesn't go away when we fix the zip_longest one:
This is a legitimate CPython bug in the lru_cache free-threading fix. The reproducer needs gc.get_objects() because only the GC's stop-the-world mechanism creates the state where the assertion detects the lock mismatch (normal single-threaded execution without GC interaction happens to work because there's no contention, but the lock is genuinely not held).
Summary of what we have — two independent bugs:
zip_longest NULL-in-ittuple — stores NULL in a GC-tracked tuple when iterators exhaust. Our patch fixes it. Worth filing standalone.
lru_cache critical section doesn't lock the cache dict — bounded_lru_cache_wrapper does Py_BEGIN_CRITICAL_SECTION(self) but _PyDict_GetItemRef_KnownHash_LockHeld asserts self->cache is locked. The gh-132641 fix converted the SetItem calls to _LockHeld variants (assuming self's CS covers the dict), but the assertion checks the dict's own mutex. Fix: change line 1498 to Py_BEGIN_CRITICAL_SECTION2(self, self->cache) (and similarly at line 1511). This is a real bug in the gh-132641 fix, reproducible only on debug free-threaded builds because release builds don't have the assertion.
change line 1498 to Py_BEGIN_CRITICAL_SECTION2(self, self->cache) (and similarly at line 1511)
oh, I didn't think about that. Makes sense yeah.
(normal single-threaded execution without GC interaction happens to work because there's no contention, but the lock is genuinely not held)
Uh, I don't think this is the case. The assertion is bypassed if the object has a refcount of 1:
lru_cache assumes it's the exclusive owner of that dict, and that assumption is not held when you have gc.get_objects(). Stop-the-world is not relevant here.
Freethreading branch merged into master