GeezerVison is showing github.com — Segfault in `_Unpickler_MemoPut` (`_pickle.c`) from `Unpickler.memo` assignment with free-threading · Issue #150505 · python/cpython
Segfault in _Unpickler_MemoPut (_pickle.c) from Unpickler.memo assignment with free-threading #150505Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
Unpickler_set_memo modifies self->memo and self->memo_size without any critical section.
When two threads set .memo on the same Unpickler instance concurrently, one can set self->memo = NULL while another is still indexing into it via _Unpickler_MemoPut, leading to a NULL-pointer dereference.
Reproducer:
import io
import pickle
from threading import Thread
shared_unpickler = pickle.Unpickler(io.BytesIO(b''))
def assign():
for _ in range(10000):
shared_unpickler.memo = {j: j for j in range(100)}
if __name__ == "__main__":
threads = [Thread(target=assign) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()TSAN report:
WARNING: ThreadSanitizer: data race (pid=214122)
Read of size 8 at 0x7dca529b9130 by thread T2:
#0 _Unpickler_MemoPut /home/cpython/./Modules/_pickle.c:1631:22 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x2122f)
#1 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7804:17 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26b77)
#2 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb)
#3 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b)
#4 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b)
#5 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca)
#6 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f)
#7 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a)
#8 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#9 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#10 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#11 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#12 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#13 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#14 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15)
#15 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
#16 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c)
#17 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8)
#18 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
#19 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8)
#20 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022)
#21 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#22 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#23 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#24 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#25 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#26 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#27 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2)
#28 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
#29 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07)
#30 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708)
#31 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb)
Previous write of size 8 at 0x7dca529b9130 by thread T1:
#0 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7816:21 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26cc4)
#1 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb)
#2 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b)
#3 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b)
#4 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca)
#5 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f)
#6 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a)
#7 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#8 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#9 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#10 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#11 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#12 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#13 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15)
#14 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
#15 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c)
#16 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8)
#17 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
#18 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8)
#19 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022)
#20 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#21 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#22 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#23 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#24 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#25 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#26 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2)
#27 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
#28 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07)
#29 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708)
#30 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb)
SUMMARY: ThreadSanitizer: data race /home/cpython/./Modules/_pickle.c:1631:22 in _Unpickler_MemoPut
==================
==214455==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7afb5f96f25a bp 0x7afb187fd6f0 sp 0x7afb187fd690 T214460)
==214455==The signal is caused by a READ memory access.
==214455==Hint: address points to the zero page.
#0 _Unpickler_MemoPut /home/cpython/./Modules/_pickle.c:1657:16 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x2125a)
#1 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7804:17 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26b77)
#2 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb)
#3 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b)
#4 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b)
#5 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca)
#6 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f)
#7 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a)
#8 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#9 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#10 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#11 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#12 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#13 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#14 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15)
#15 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
#16 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c)
#17 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8)
#18 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
#19 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8)
#20 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022)
#21 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463)
#22 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
#23 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013)
#24 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11)
#25 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
#26 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132)
#27 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2)
#28 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
#29 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07)
#30 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708)
#31 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb)
#32 __tsan_thread_start_func <null> (python3.16t+0xf9182)
#33 start_thread nptl/pthread_create.c:447:8 (libc.so.6+0x9caa3) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
#34 clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 (libc.so.6+0x129c6b) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV /home/cpython/./Modules/_pickle.c:1657:16 in _Unpickler_MemoPut
==214455==ABORTING
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activitykumaraditya303 commented on May 28, 2026 I think there are other issues as well if pickler objects are shared across threads, is this actually supposed to work? AhmedMagdy9876 commented on May 28, 2026 @kumaraditya303 the fix addresses the specific race in memo assignment, not Unpickler's other mutable state. thay would also race if the object were shared across threads during .load().
The question of whether sharing an Unpickler across threads shall be a supported use case seems worth settling first. If the answer is no, a documentation note and/or annotation would be more appropriate than scattered locking. If the answer is yes, then a more comprehensive audit of all mutable fields would be needed beyond just memo.
What do you think on the intended thread-safety contract for Unpickler? Happy to adjust the approach based on that. kumaraditya303 commented on May 28, 2026 I don't think pickler objects are expected to work across threads, @serhiy-storchaka what do you think? serhiy-storchaka commented on May 28, 2026 I do not see what can be a reason of sharing pickler or unpickler objects between threads. Neither dump() nor load() can work concurrently, because they change the internal state (memo) depending on input data and timing, and the order of changes matters, even if we make them atomic. devdanzin commented on Jul 23, 2026 Corroborating @kumaraditya303's "there are other issues as well if pickler objects are shared across threads" — the Pickler side has the same class of problem (this issue is the Unpickler _Unpickler_MemoPut). A shared Pickler takes no per-object critical section, so concurrent dump() / clear_memo() crash it: PyMemoTable_Clear does Py_XDECREF(me_key) + memset over mt_table (Modules/_pickle.c:816-819) with no lock, racing _PyMemoTable_Lookup / PyMemoTable_Size reads (:848 / :807, via Pickler.dump()), plus the shared output buffer (_Pickler_Write :1106). The result is a double-DECREF of a memo'd key (_Py_NegativeRefcount) and a _Pickler_Write SIGSEGV.
Reproducer (free-threaded build, PYTHON_GIL=0) — crashes 6/6 on a plain free-threaded debug build:
import _pickle, io, threading
p = _pickle.Pickler(io.BytesIO())
N = 30000
def dumper():
for _ in range(N):
try: p.dump([1, 2, 3, "x", {"k": 1}])
except Exception: pass
def clearer():
for _ in range(N):
try: p.clear_memo()
except Exception: pass
ts = [threading.Thread(target=dumper) for _ in range(4)] + \
[threading.Thread(target=clearer) for _ in range(4)]
for t in ts: t.start()
for t in ts: t.join()Given @serhiy-storchaka's and @kumaraditya303's point that sharing pickler/unpickler objects across threads isn't a scenario that can work (both dump() and load() mutate the memo), we're not filing this separately — just recording it here so the Pickler-side "other issues" are documented next to the Unpickler.memo fix, in case it's useful when deciding how far to take the hardening.
Found with fusil's --tsan mode; reproducer reduced with AI assistance (Claude Code), verified by hand on free-threaded debug/TSan builds of main.
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error StatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|