GeezerVison is showing github.com — Free-threading race betwen two Py_SETREF · Issue #161714 · python/cpython
Free-threading race betwen two Py_SETREF #161714Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
This can be triggered with existing Python objects that use Py_SETREF or Py_XSETREF such as Exceptions (although slightly contrived):
import threading
e = Exception()
def worker():
for _ in range(1_000_000):
e.__init__(1)
threads = [threading.Thread(target=worker) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()Gives on a 3.16 free-threaded debug build:
../cpython/Include/refcount.h:617: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback
object address : 0x2000204c3b0
object refcount : 0
object type : 0x5623737ad5c8
object type name: tuple
object repr : <refcnt 0 at 0x2000204c3b0>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Stack (most recent call first):
File "/<home>/projects/python/setref_bug.py", line 8 in worker
File "/<home>/projects/python/cpython/Lib/threading.py", line 1161 in run
File "/<home>/projects/python/cpython/Lib/threading.py", line 1219 in _bootstrap_inner
File "/<home>/projects/python/cpython/Lib/threading.py", line 1181 in _bootstrap
[1] 397225 abort (core dumped) ./build/python setref_bug.py
This arrives because Py_SETREF copies the ref and stores a new new one as two separate non-atomic steps. So two threads can do:
T1: old = *slot
T2: old = *slot // same object as T1
T1: *slot = a
T2: *slot = b // a's reference is lost, so a leaks
T1: Py_XDECREF(old)
T2: Py_XDECREF(old) // the slot's one reference to old is dropped twice
I ran into this trying to build a cache which swapped an old value for a new value with Py_SETREF and finding it sometimes resulted in a double free when two threads swapped the value simultaneously (And ended up thinking they had the last reference).
CPython versions tested on:
3.16, CPython main branch
Operating systems tested on:
Linux Activitycolesbury commented on Nov 9, 2026 Concurrent Py_SETREF(dst, src) / Py_XSETREF calls on the same dst are not intended to be thread-safe.
Separately, we also generally don't try to make concurrent __init__() on the same object thread-safe unless there's some code in the wild that actually does this. We may revisit that in the future, but for now, it seems like it would add code complexity and performance regressions for little benefit. cmaloney commented on Nov 10, 2026 ContributorAuthor More actions I'm okay with that as a documented limitation.
Having an atomic version that is opt-in would make lazy thread-safe caches significantly easier to write: init to Py_None or NULL then once make a cache dictionary just Py_SETREF_THREADSAFE or Py_XSETREF_THREADSAFE. Open to a helper like that?
Across CPython currently there's quite a few descriptors which individual elements use these (ex. property___set_name___impl) where locking the whole object for the single setter likely doesn't make sense but having two threads setting .name concurrently not crash the interpreter would be nice.
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|