GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — data race when resizing unicode objects · Issue #132070 · python/cpython

data race when resizing unicode objects #132070

New issue

Copy link

New issue

Copy link

Closed

data race when resizing unicode objects#132070

Copy link

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtopic-unicodetype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

kumaraditya303

opened

on Apr 4, 2025

Issue body actions

When running ctypes tests in parallel, the following data race is detected:

env TSAN_OPTIONS="suppressions=Tools/tsan/suppressions_free_threading.txt halt_on_error=1" ./python -m test test_ctypes --parallel-threads 4

==================
WARNING: ThreadSanitizer: data race (pid=19492)
  Atomic read of size 8 at 0x7f1b0a230680 by thread T1323:
    #0 _Py_atomic_load_uintptr_relaxed /home/realkumaraditya/cpython/./Include/cpython/pyatomic_gcc.h:375:10 (python+0x271055) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #1 _Py_IsOwnedByCurrentThread /home/realkumaraditya/cpython/./Include/object.h:254:12 (python+0x271055)
    #2 _Py_TryIncrefFast /home/realkumaraditya/cpython/./Include/internal/pycore_object.h:558:9 (python+0x271055)
    #3 _Py_TryIncrefCompare /home/realkumaraditya/cpython/./Include/internal/pycore_object.h:597:9 (python+0x271055)
    #4 compare_unicode_unicode_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1432:18 (python+0x271055)
    #5 do_lookup /home/realkumaraditya/cpython/Objects/dictobject.c:1013:23 (python+0x271055)
    #6 unicodekeys_lookup_unicode_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1449:12 (python+0x271055)
    #7 _Py_dict_lookup_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1504:18 (python+0x270757) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #8 _PyDict_GetItemRef_KnownHash /home/realkumaraditya/cpython/Objects/dictobject.c:2353:21 (python+0x272c0e) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #9 PyDict_GetItemRef /home/realkumaraditya/cpython/Objects/dictobject.c:2389:12 (python+0x272c0e)
    #10 cache_struct_converter /home/realkumaraditya/cpython/./Modules/_struct.c:2524:9 (_struct.cpython-314t-x86_64-linux-gnu.so+0x5338) (BuildId: cd844f32564aba7a2b18bdb1e258a78c397c8ebc)
    #11 calcsize /home/realkumaraditya/cpython/./Modules/clinic/_struct.c.h:266:10 (_struct.cpython-314t-x86_64-linux-gnu.so+0x4ac7) (BuildId: cd844f32564aba7a2b18bdb1e258a78c397c8ebc)
    #12 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2225:35 (python+0x3fef50) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #13 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #14 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #15 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #16 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f66b0) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #17 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:94:18 (python+0x1f66b0)
    #18 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #19 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #20 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #21 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2411:32 (python+0x3ff399) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #22 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #23 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #24 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #25 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f65ff) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #26 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:72:20 (python+0x1f65ff)
    #27 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #28 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #29 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #30 thread_run /home/realkumaraditya/cpython/./Modules/_threadmodule.c:353:21 (python+0x5998d2) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #31 pythread_wrapper /home/realkumaraditya/cpython/Python/thread_pthread.h:242:5 (python+0x4f4157) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)

  Previous write of size 8 at 0x7f1b0a230680 by thread T1321:
    #0 __tsan_memcpy <null> (python+0xdd10f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #1 _mi_memcpy /home/realkumaraditya/cpython/./Include/internal/mimalloc/mimalloc/internal.h:929:3 (python+0x2af9bf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #2 _mi_heap_realloc_zero /home/realkumaraditya/cpython/Objects/mimalloc/alloc.c:745:7 (python+0x2af9bf)
    #3 mi_heap_realloc /home/realkumaraditya/cpython/Objects/mimalloc/alloc.c:753:10 (python+0x2cce30) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #4 _PyObject_MiRealloc /home/realkumaraditya/cpython/Objects/obmalloc.c:276:12 (python+0x2cce30)
    #5 PyObject_Realloc /home/realkumaraditya/cpython/Objects/obmalloc.c:1416:12 (python+0x2cfc12) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #6 resize_compact /home/realkumaraditya/cpython/Objects/unicodeobject.c:1178:31 (python+0x359915) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #7 _PyUnicodeWriter_Finish /home/realkumaraditya/cpython/Objects/unicodeobject.c:14157:16 (python+0x33162a) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #8 PyUnicode_Format /home/realkumaraditya/cpython/Objects/unicodeobject.c:15361:12 (python+0x35e728) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #9 unicode_mod /home/realkumaraditya/cpython/Objects/unicodeobject.c:14332:12 (python+0x36df71) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #10 binary_op1 /home/realkumaraditya/cpython/Objects/abstract.c:964:13 (python+0x1bdb83) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #11 binary_op /home/realkumaraditya/cpython/Objects/abstract.c:1003:24 (python+0x1be2a8) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #12 PyNumber_Remainder /home/realkumaraditya/cpython/Objects/abstract.c:1187:1 (python+0x1be2a8)
    #13 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:62:35 (python+0x3f7ae5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #14 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #15 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #16 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #17 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f66b0) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #18 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:94:18 (python+0x1f66b0)
    #19 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #20 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #21 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #22 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2411:32 (python+0x3ff399) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #23 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #24 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #25 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #26 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f65ff) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #27 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:72:20 (python+0x1f65ff)
    #28 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #29 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #30 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #31 thread_run /home/realkumaraditya/cpython/./Modules/_threadmodule.c:353:21 (python+0x5998d2) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #32 pythread_wrapper /home/realkumaraditya/cpython/Python/thread_pthread.h:242:5 (python+0x4f4157) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)

Linked PRs

Activity

kumaraditya303
added
topic-free-threading
on Apr 4, 2025
picnixz
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

topic-unicode
on Apr 4, 2025

kumaraditya303 commented on Apr 4, 2025

kumaraditya303

on Apr 4, 2025

ContributorAuthor

More actions

I think in free-threading Py_REFCNT(op)==1 should be replaced by PyObject_IsUniquelyReferenced in such functions in unicodeobject.c:

cpython/Objects/unicodeobject.c

Lines 1197 to 1202 in a126cef

	static int
	resize_inplace(PyObject *unicode, Py_ssize_t length)
	{
	assert(!PyUnicode_IS_COMPACT(unicode));
	assert(Py_REFCNT(unicode) == 1);

colesbury commented on Apr 4, 2025
colesbury
on Apr 4, 2025
Contributor
More actions
We should leave the assertions as is. I don't think that's related to this.

The race is because PyObject_Realloc() calls memcpy to fill in the new allocation, if the existing allocation can't be reused. The memcpy doesn't use relaxed atomic writes, so it can race with an one of the optimistic, non-locking reads.

For now, we should just add a suppression.
colesbury commented on Apr 4, 2025
colesbury
on Apr 4, 2025
Contributor
More actions
We should change Py_REFCNT(op)==1 to PyObject_IsUniquelyReferenced for places where it controls behaviors, such as in if-statements, but the uses in assertions don't really matter.
kumaraditya303
added a commit that references this issue on Apr 16, 2025
gh-132070: add `PyObject_Realloc` suppression in free-threading (#132468
4b15d10
godlygeek commented on Apr 27, 2025
godlygeek
on Apr 27, 2025
Contributor
More actions
Would it make sense to change the behavior of Py_REFCNT in free-threading so that Py_REFCNT(op)==1 remains a valid check? I think that due to TOCTOU the only thing an extension module can usefully do with Py_REFCNT even in GIL builds is check whether the recount of an object is or isn't 1. If you're no longer able to use Py_REFCNT to check that in FT builds, it seems like Py_REFCNT becomes totally useless, and it would be much more useful to do

#define Py_REFCNT(x) (PyObject_IsUniquelyReferenced((x)) ? 1 : INT_MAX)

or some other arbitrary large number.
serhiy-storchaka commented on Apr 28, 2025
serhiy-storchaka
on Apr 28, 2025
Member
More actions
There are some places where Py_REFCNT is compared with 2 or 0.
Reacted by Sam Gross
godlygeek commented on Apr 28, 2025
godlygeek
on Apr 28, 2025
Contributor
More actions
Sure, but those don't work in free threaded builds today, right? So my question is whether we can make it so that Py_REFCNT in an FT build returns something wrong but usually useful instead of something wrong and never useful.
ZeroIntensity commented on Apr 28, 2025
ZeroIntensity
on Apr 28, 2025
Member
More actions
Why couldn't it just be _PyObject_IsUniquelyReferenced(op) ? 1 : actual_refcnt(op)?
colesbury commented on Apr 28, 2025
colesbury
on Apr 28, 2025
Contributor
More actions
@ZeroIntensity - that's identical to actual_refcnt(op). _PyObject_IsUniquelyReferenced(1) == 1 implies Py_REFCNT(op) == 1. The inverse is not necessarily true.

I agree with @serhiy-storchaka here. Py_REFCNT() is used for lots of things. Breaking it for a wide variety of use cases in order to try to make the _PyObject_IsUniquelyReferenced() use case more robust is not worth it.
ZeroIntensity commented on Apr 28, 2025
ZeroIntensity
on Apr 28, 2025
Member
More actions

The inverse is not necessarily true.

Oh, that makes sense.

Is this documented anywhere? The docs do say that you can check for a reference count of 1. If that has problems with FT, we need to update it.

We should probably expose _PyObject_IsUniquelyReferenced as an unstable API for this then. Is there any current plan for that?

Reacted by Matt Wozniski and Donghee Na
corona10
added a commit that references this issue on Apr 29, 2025
gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (gh-13…
75cbb8d
miss-islington
added a commit that references this issue on Apr 29, 2025
pythongh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (p…
4f40bf2
corona10
added a commit that references this issue on Apr 29, 2025
[3.13] gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (…
7ffef8d
corona10 commented on Apr 29, 2025
corona10
on Apr 29, 2025
Member
More actions
@ZeroIntensity Would you like to open a new issue to track adding unstable C API?
Reacted by Peter Bierma
corona10
closed this as completed on Apr 29, 2025
colesbury
added 2 commits that reference this issue on Jan 5, 2026
pythongh-132070: Fix PyObject_Realloc thread-safety in free threaded …
a4c76a6
gh-132070: Fix PyObject_Realloc thread-safety in free threaded Python (…
98e55d7
thunder-coding
added a commit that references this issue on Feb 15, 2026
pythongh-132070: Fix PyObject_Realloc thread-safety in free threaded …
140f221

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

data race when resizing unicode objects #132070

Description

Linked PRs

Activity

kumaraditya303 commented on Apr 4, 2025

colesbury commented on Apr 4, 2025

colesbury commented on Apr 4, 2025

godlygeek commented on Apr 27, 2025

serhiy-storchaka commented on Apr 28, 2025

godlygeek commented on Apr 28, 2025

ZeroIntensity commented on Apr 28, 2025

colesbury commented on Apr 28, 2025

ZeroIntensity commented on Apr 28, 2025

corona10 commented on Apr 29, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions