GeezerVison is showing github.com — free threading: struct iter_unpack is not memory safe · Issue #154013 · python/cpython
free threading: struct iter_unpack is not memory safe #154013DescriptionBug report
Bug description:
NTHREADS = 8
ROUNDS = 5000
S = struct.Struct("i")
BUF = b"\x00\x00\x00\x00" * 500 # 500 "i" items; small so it exhausts fast
def drain(it, barrier):
barrier.wait()
for _ in it:
pass
def main():
for _ in range(ROUNDS):
it = S.iter_unpack(BUF)
barrier = threading.Barrier(NTHREADS)
threads = [threading.Thread(target=drain, args=(it, barrier))
for _ in range(NTHREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
print("survived this run")Structs have come up before #149816 (comment) with some WF issues related to structs although they are focused around initialising as far as I can see.
TSAN
WARNING: ThreadSanitizer: data race (pid=7225)
Write of size 8 at 0x000112338508 by thread T4:
#0 unpackiter_iternext _struct.c:2278 (_struct.cpython-316td-darwin.so:arm64+0xa274)
#1 _PyForIter_VirtualIteratorNext ceval.c:3775 (python.exe:arm64+0x100391868)
#2 _PyEval_EvalFrameDefault generated_cases.c.h:6352 (python.exe:arm64+0x10037276c)
#3 _PyEval_Vector ceval.c:2172 (python.exe:arm64+0x100359070)
#4 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000edce8)
#5 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#6 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x1000ef4d4)
#7 method_vectorcall classobject.c:55 (python.exe:arm64+0x1000f1fd8)
#8 context_run context.c:731 (python.exe:arm64+0x1003c9298)
#9 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x100109bcc)
#10 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#11 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000ed7ac)
#12 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:768 (python.exe:arm64+0x100359bb8)
#13 _PyEval_EvalFrameDefault generated_cases.c.h:1906 (python.exe:arm64+0x10036338c)
#14 _PyEval_Vector ceval.c:2172 (python.exe:arm64+0x100359070)
#15 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000edce8)
#16 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#17 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x1000ef4d4)
#18 method_vectorcall classobject.c:55 (python.exe:arm64+0x1000f1fd8)
#19 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000ed654)
#20 _PyObject_Call call.c:348 (python.exe:arm64+0x1000ed89c)
#21 PyObject_Call call.c:373 (python.exe:arm64+0x1000edaac)
#22 thread_run _threadmodule.c:388 (python.exe:arm64+0x1005c0474)
#23 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x1004c68a4)
Previous write of size 8 at 0x000112338508 by thread T3:
#0 unpackiter_iternext _struct.c:2278 (_struct.cpython-316td-darwin.so:arm64+0xa274)
#1 _PyForIter_VirtualIteratorNext ceval.c:3775 (python.exe:arm64+0x100391868)
#2 _PyEval_EvalFrameDefault generated_cases.c.h:6352 (python.exe:arm64+0x10037276c)
#3 _PyEval_Vector ceval.c:2172 (python.exe:arm64+0x100359070)
#4 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000edce8)
#5 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#6 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x1000ef4d4)
#7 method_vectorcall classobject.c:55 (python.exe:arm64+0x1000f1fd8)
#8 context_run context.c:731 (python.exe:arm64+0x1003c9298)
#9 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x100109bcc)
#10 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#11 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000ed7ac)
#12 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:768 (python.exe:arm64+0x100359bb8)
#13 _PyEval_EvalFrameDefault generated_cases.c.h:1906 (python.exe:arm64+0x10036338c)
#14 _PyEval_Vector ceval.c:2172 (python.exe:arm64+0x100359070)
#15 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000edce8)
#16 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1000ebe60)
#17 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x1000ef4d4)
#18 method_vectorcall classobject.c:55 (python.exe:arm64+0x1000f1fd8)
#19 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000ed654)
#20 _PyObject_Call call.c:348 (python.exe:arm64+0x1000ed89c)
#21 PyObject_Call call.c:373 (python.exe:arm64+0x1000edaac)
#22 thread_run _threadmodule.c:388 (python.exe:arm64+0x1005c0474)
#23 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x1004c68a4)
SUMMARY: ThreadSanitizer: data race _struct.c:2270 in unpackiter_iternext
==================
ThreadSanitizer:DEADLYSIGNAL
==7225==ERROR: ThreadSanitizer: SEGV on unknown address 0xffffabbbbbbbbd20 (pc 0x000103d5cf2c bp 0x00016d179400 sp 0x00016d1793f0 T32990319)
==7225==The signal is caused by a READ memory access.
#0 OUTLINED_FUNCTION_10 <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x78f2c)
#1 _PyType_GetDict typeobject.c:542 (python.exe:arm64+0x10021f68c)
#2 _Py_Specialize_LoadAttr specialize.c:1006 (python.exe:arm64+0x10049fe58)
#3 _PyEval_EvalFrameDefault generated_cases.c.h:8671 (python.exe:arm64+0x100379ee0)
#4 _PyEval_Vector ceval.c:2172 (python.exe:arm64+0x100359070)
#5 PyEval_EvalCode ceval.c:679 (python.exe:arm64+0x100358954)
#6 run_eval_code_obj pythonrun.c:1406 (python.exe:arm64+0x10049bc44)
#7 run_mod pythonrun.c:1509 (python.exe:arm64+0x10049b7f4)
#8 _PyRun_File pythonrun.c:1332 (python.exe:arm64+0x1004975a4)
#9 _PyRun_SimpleFile pythonrun.c:544 (python.exe:arm64+0x100495b78)
#10 _PyRun_AnyFile pythonrun.c:92 (python.exe:arm64+0x1004953d8)
#11 pymain_run_file main.c:494 (python.exe:arm64+0x1004e320c)
#12 Py_RunMain main.c:891 (python.exe:arm64+0x1004e270c)
#13 pymain_main main.c:921 (python.exe:arm64+0x1004e2c70)
#14 Py_BytesMain main.c:945 (python.exe:arm64+0x1004e2d1c)
#15 main python.c:15 (python.exe:arm64+0x100000a64)
#16 start <null> (dyld:arm64e+0x8d50)
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs
Activitypicnixz commented on Jul 18, 2026 Please, include the ASAN/TSAN traceback if any, or a crash traceback if any. ZeroIntensity commented on Jul 18, 2026 In general, I think we should avoid adding synchronization to iterator types. A few cases have slipped through (primarily stuff in itertools), but we've rejected bug reports on this in the past because:
- Fixing it slows down the single-threaded case and complicates the code.
- There are very few cases where one might want to use an iterator concurrently, and I can't imagine any case for
Struct iterators.
This is why we have threading.serialize_iterator in 3.15.
Unless you have a compelling use case for this, let's avoid doing this for now. johng commented on Jul 19, 2026 ContributorAuthor More actions @ZeroIntensity understand what you are saying, however believe I was able to add the thread safety without any additional overhead. Would you kindly be able to have a look at the PR above, thank you! ZeroIntensity commented on Jul 19, 2026 Overhead is just one aspect. Looking at the PR, it adds some significant complexity using atomics that are difficult to reason about.
I'm going to close this. We might revisit iterator types in the future, but they're not important for the overall free-threading effort, and we shouldn't encourage people to scour the codebase looking for thread-safety issues with them. johng commented on Jul 19, 2026 ContributorAuthor More actions Are there any plans to make parallel iterator access explicitly discouraged or even blocked in that case? Even if the use case is uncommon (for which I agree) leaving in clear ways users can cause data corruption or interpreter crashes seems worrying.
To add I'm not saying we should encourage the workflows but maybe have a better UX where it's explicitly dis-allowed instead (or even documented) ZeroIntensity commented on Jul 19, 2026 We want to fix concurrent iteration for cases where it's useful. If a user legitimately found a crash via concurrent iteration, that's a strong sign that an iterator type should be made thread-safe.
leaving in clear ways users can cause data corruption or interpreter crashes seems worrying.
I think you would be surprised at how many things can crash the interpreter, even in the normal GILful mode. Some things are fixable, but they're not worth the time, the potential performance hit, or the additional maintenance burden, because nobody does it in practice. Practicality beats purity!
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|