GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Data race between compare_generic and insert_combined_dict under free-threading · Issue #132641 · python/cpython

Data race between compare_generic and insert_combined_dict under free-threading #132641

New issue

Copy link

New issue

Copy link

Closed

#133787

Closed

Data race between compare_generic and insert_combined_dict under free-threading#132641

#133787

Copy link

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

vfdev-5

opened

on Apr 17, 2025

Issue body actions

Bug report

Bug description:

I built main branch and observed the following races under free-threading in cpython 3.14 (Python 3.14.0a7+ experimental free-threading build (heads/main:e42bda94411, Apr 17 2025, 14:08:39) [Clang 18.1.3 (1ubuntu1)])

Race 1

WARNING: ThreadSanitizer: data race (pid=33872)
  Read of size 8 at 0x7fffc22d7ad8 by thread T5 (mutexes: read M0):
    #0 compare_generic /project/cpython/Objects/dictobject.c:1107:13 (python3.14+0x26edbc) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #1 do_lookup /project/cpython/Objects/dictobject.c:1013:23 (python3.14+0x26edbc)
    #2 dictkeys_generic_lookup /project/cpython/Objects/dictobject.c:1132:12 (python3.14+0x26edbc)
    #3 _Py_dict_lookup /project/cpython/Objects/dictobject.c:1298:14 (python3.14+0x26edbc)
    #4 _PyDict_GetItemRef_KnownHash_LockHeld /project/cpython/Objects/dictobject.c:2330:21 (python3.14+0x272a14) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
   ...

  Previous atomic write of size 8 at 0x7fffc22d7ad8 by thread T6 (mutexes: read M0):
    #0 _Py_atomic_store_ptr_release /project/cpython/./Include/cpython/pyatomic_gcc.h:565:3 (python3.14+0x284d66) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #1 insert_combined_dict /project/cpython/Objects/dictobject.c:1748:9 (python3.14+0x284d66)
    #2 insertdict /project/cpython/Objects/dictobject.c:1854:13 (python3.14+0x274254) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #3 _PyDict_SetItem_KnownHash_LockHeld /project/cpython/Objects/dictobject.c:2656:12 (python3.14+0x273be9) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #4 _PyDict_SetItem_KnownHash /project/cpython/Objects/dictobject.c:2673:11 (python3.14+0x274635) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
  ...

Race 2

WARNING: ThreadSanitizer: data race (pid=33872)
  Read of size 8 at 0x7fffc22d7ad0 by thread T5 (mutexes: read M0):
    #0 compare_generic /project/cpython/Objects/dictobject.c:1110:13 (python3.14+0x26edd5) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #1 do_lookup /project/cpython/Objects/dictobject.c:1013:23 (python3.14+0x26edd5)
    #2 dictkeys_generic_lookup /project/cpython/Objects/dictobject.c:1132:12 (python3.14+0x26edd5)
    #3 _Py_dict_lookup /project/cpython/Objects/dictobject.c:1298:14 (python3.14+0x26edd5)
    #4 _PyDict_GetItemRef_KnownHash_LockHeld /project/cpython/Objects/dictobject.c:2330:21 (python3.14+0x272a14) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
   ...

  Previous atomic write of size 8 at 0x7fffc22d7ad0 by thread T6 (mutexes: read M0):
    #0 insert_combined_dict /project/cpython/Objects/dictobject.c (python3.14+0x284d8c) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #1 insertdict /project/cpython/Objects/dictobject.c:1854:13 (python3.14+0x274254) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #2 _PyDict_SetItem_KnownHash_LockHeld /project/cpython/Objects/dictobject.c:2656:12 (python3.14+0x273be9) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    #3 _PyDict_SetItem_KnownHash /project/cpython/Objects/dictobject.c:2673:11 (python3.14+0x274635) (BuildId: d491981d76fd0b67bcb5e01a978d07da019800b8)
    ...

Full report: https://gist.github.com/vfdev-5/cbb9189043737d023b755191b62951cf

cc @hawkinsp

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

vfdev-5
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
on Apr 17, 2025
picnixz
added
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

topic-free-threading
on Apr 17, 2025
thomashammett commented on Apr 18, 2025
thomashammett
on Apr 18, 2025
More actions
I'd like to work on this issue and submit a PR to fix it. Is anyone already working on this, or may I take it? Thank you.
vstinner commented on Apr 23, 2025
vstinner
on Apr 23, 2025
Member
More actions

I built main branch and observed the following races under free-threading in cpython 3.14

Which code did you run to trigger these data races?
vfdev-5 commented on Apr 23, 2025
vfdev-5
on Apr 23, 2025
ContributorAuthor
More actions
It is from JAX TSAN CI job running cpu tests suite. I can try to get a small reproducer.
hawkinsp commented on May 9, 2025
hawkinsp
on May 9, 2025
Contributor
More actions
We are still seeing this frequently in our CI.

I'm trying to come up with a reproducer for this and so far haven't succeeded. However, reading the code, I'm curious about the following:

In bounded_lru_cache_get_lock_held we hold a critical section of self, i.e., the lru cache object itself:

cpython/Modules/_functoolsmodule.c

Line 1482 in 6d5a8c2

Py_BEGIN_CRITICAL_SECTION(self);

We then use that critical section to call _PyDict_GetItemRef_KnownHash_LockHeld with no other locking

cpython/Modules/_functoolsmodule.c

Line 1309 in 6d5a8c2

int res = _PyDict_GetItemRef_KnownHash_LockHeld((PyDictObject *)self->cache, key_, hash_,

. In other words, the read part of the cache seems to assume that the critical section on the lru_cache object itself protects the dictionary state.

However on the write side of the cache, we start by acquiring a critical section on self as well (

cpython/Modules/_functoolsmodule.c

Line 1500 in 6d5a8c2

result = bounded_lru_cache_update_lock_held(self, result, key, hash);

). However in some code paths we call _PyDict_SetItem_KnownHash (e.g.,

cpython/Modules/_functoolsmodule.c

Line 1386 in 6d5a8c2

if (_PyDict_SetItem_KnownHash(self->cache, key, (PyObject *)link,

), which acquires a critical section on the dictionary object (

cpython/Objects/dictobject.c

Line 2693 in 6d5a8c2

Py_BEGIN_CRITICAL_SECTION(op);

), which I think we should think of as implicitly releasing the outer lock.

In other words, we are confused about which critical section is protecting the dictionary state. Is it the critical section on the lru cache itself, or it the critical section on the dict it contains?
hawkinsp
added 9 commits that reference this issue on May 9, 2025
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
11867b9
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
5b006e2
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
b547cb3
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
9840f69
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
532e3e7
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
e611a03
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
a7f785c
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
75aa17f
pythongh-132641: Fix race in lru_cache due to inconsistent critical s…
50db857
kumaraditya303
closed this as completedin #133787 on May 13, 2025
kumaraditya303
added a commit that references this issue on May 13, 2025
gh-132641: fix race in `lru_cache` under free-threading (#133787)
9ad0c7b
miss-islington
added a commit that references this issue on May 13, 2025
pythongh-132641: fix race in `lru_cache` under free-threading (python…
1594c26
kumaraditya303
added a commit that references this issue on May 14, 2025
[3.14] gh-132641: fix race in `lru_cache` under free-threading (GH-13…
66d6860
Pranjal095
added a commit that references this issue on Jul 12, 2025
pythongh-132641: fix race in `lru_cache` under free-threading (python…
d0405cc
taegyunkim
added a commit that references this issue on Aug 4, 2025
pythongh-132641: fix race in `lru_cache` under free-threading (python…
a008ae6

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Data race between compare_generic and insert_combined_dict under free-threading #132641

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

thomashammett commented on Apr 18, 2025

vstinner commented on Apr 23, 2025

vfdev-5 commented on Apr 23, 2025

hawkinsp commented on May 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions