GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads · Issue #140257 · python/cpython

ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads #140257

New issue

Copy link

New issue

Copy link

Closed

#140294

Closed

ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads#140257

#140294

Copy link

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

ashm-dev

opened

on Oct 17, 2025

Issue body actions

Bug report

Bug description:

ThreadSanitizer detects a data race between the main thread clearing the interpreter during finalization and daemon threads attempting to acquire the GIL. The race occurs on _PyRuntime.ceval.eval_breaker at address that is accessed by both:

Main thread (write): interpreter_clear() → writes to eval_breaker
Daemon thread T7 (atomic write): take_gil() → _Py_unset_eval_breaker_bit() → atomic operation on the same address

This appears to be related to but distinct from #124878, as the race involves interpreter_clear rather than free_threadstate, and occurs during take_gil rather than drop_gil.

How to Reproduce

Build configuration:

CC=clang CXX=clang++ ./configure --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes --disable-optimizations --with-lto=full

make -j$(nproc)

Run command:

./python -X dev -X showrefcount bug.py

Test script (bug.py):

import threading
from functools import lru_cache

@lru_cache(None)
def fib(n):
    """Function that JIT will compile"""
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

def hammer_jit():
    """Call the function many times to force JIT compilation"""
    for i in range(1_000_000):
        fib(i)

# Create a race: all threads simultaneously force JIT compilation
threads = [threading.Thread(target=hammer_jit) for _ in range(16)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Output

Details

➜  main git:(main) ✗ ./python -X dev -X showrefcount bug.py
^C^CException ignored in the internal traceback machinery:
^CTraceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1379, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1341, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1267, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1304, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1278, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1410, in find_spec
  File "<frozen importlib._bootstrap_external>", line 166, in _path_isfile
  File "<frozen importlib._bootstrap_external>", line 158, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 152, in _path_stat
KeyboardInterrupt:
Traceback (most recent call last):
  File "/home/shamil/oss/cpython/main/bug.py", line 18, in <module>
    for t in threads: t.start()
  File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1126, in join
^Cobject address  : 0x7ffff5d41c50
object refcount : 3
object type     : 0x555556087868
object type name: KeyboardInterrupt
object repr     : KeyboardInterrupt()
lost sys.stderr
^CException ignored on threading shutdown:
Traceback (most recent call last):
  File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1577, in _shutdown
    _thread_shutdown()
    ~~~~~~~~~~~~~~~~^^
KeyboardInterrupt
==================
WARNING: ThreadSanitizer: data race (pid=22017)
  Write of size 8 at 0x55555611abc8 by main thread:
    #0 interpreter_clear /home/shamil/oss/cpython/main/Python/pystate.c:769:26 (python+0x6cd57a) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 _PyInterpreterState_Clear /home/shamil/oss/cpython/main/Python/pystate.c:904:5 (python+0x699f80) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #2 finalize_interp_clear /home/shamil/oss/cpython/main/Python/pylifecycle.c:1920:5 (python+0x699f80)
    #3 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #4 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #5 Py_FinalizeEx /home/shamil/oss/cpython/main/Python/pylifecycle.c:2378:12 (python+0x72569b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #6 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:774:9 (python+0x72569b)
    #7 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #9 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145)
    #11 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)            #12 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)

  Previous atomic write of size 8 at 0x55555611abc8 by thread T7:
    #0 _Py_atomic_and_uintptr /home/shamil/oss/cpython/main/./Include/cpython/pyatomic_gcc.h:218:10 (python+0x5c6cbe) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 _Py_unset_eval_breaker_bit /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:350:5 (python+0x5c6cbe)
    #2 take_gil /home/shamil/oss/cpython/main/Python/ceval_gil.c:351:21 (python+0x5c6cbe)
    #3 _PyEval_AcquireLock /home/shamil/oss/cpython/main/Python/ceval_gil.c:588:5 (python+0x6d152e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #4 _PyThreadState_Attach /home/shamil/oss/cpython/main/Python/pystate.c:2100:9 (python+0x6d152e)
    #5 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)             #6 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #7 check_periodics /home/shamil/oss/cpython/main/Python/ceval_macros.h:412:16 (python+0x53b92b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:10296:31 (python+0x53b92b)
    #9 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #11 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)                  #12 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #13 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)          #14 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x586544) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)                                                                                                                                                    #15 context_run /home/shamil/oss/cpython/main/Python/context.c:728:29 (python+0x586544)
    #16 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3710:35 (python+0x5139a5) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #17 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #18 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #19 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #20 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #21 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #22 _PyVectorcall_Call /home/shamil/oss/cpython/main/Objects/call.c:273:16 (python+0x2854fd) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #23 _PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:348:16 (python+0x285764) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #24 PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:373:12 (python+0x828572) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #25 thread_run /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:387:21 (python+0x828572)
    #26 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #27 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)

  Location is global '_PyRuntime' of size 359232 at 0x5555560c3408 (python+0xbc6bc8)
  Thread T7 'Thread-7 (hamme' (tid=22032, running) created by main thread at:                                                                                      #0 pthread_create <null> (python+0xfc21e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #1 do_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:281:14 (python+0x709041) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #2 PyThread_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:323:9 (python+0x828011) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #3 ThreadHandle_start /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:474:9 (python+0x828011)                                                          #4 do_start_new_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:1920:9 (python+0x828011)
    #5 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #6 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #7 cfunction_call /home/shamil/oss/cpython/main/Objects/methodobject.c:564:18 (python+0x354c1b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #8 _PyObject_MakeTpCall /home/shamil/oss/cpython/main/Objects/call.c:242:18 (python+0x283b90) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #9 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:167:16 (python+0x2838b4) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #10 PyObject_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c:327:12 (python+0x51093d) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #11 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3188:35 (python+0x51093d)
    #12 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #13 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #14 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #15 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7)
    #16 PyEval_EvalCode /home/shamil/oss/cpython/main/Python/ceval.c:884:21 (python+0x4f9e99) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #17 run_eval_code_obj /home/shamil/oss/cpython/main/Python/pythonrun.c:1365:12 (python+0x6d7f60) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #18 run_mod /home/shamil/oss/cpython/main/Python/pythonrun.c:1459:19 (python+0x6d7f60)
    #19 pyrun_file /home/shamil/oss/cpython/main/Python/pythonrun.c:1293:15 (python+0x6d6adb) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #20 _PyRun_SimpleFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:521:13 (python+0x6d6adb)
    #21 _PyRun_AnyFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:81:15 (python+0x6d5736) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #22 pymain_run_file_obj /home/shamil/oss/cpython/main/Modules/main.c:410:15 (python+0x727676) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #23 pymain_run_file /home/shamil/oss/cpython/main/Modules/main.c:429:15 (python+0x727676)
    #24 pymain_run_python /home/shamil/oss/cpython/main/Modules/main.c:691:21 (python+0x72634e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #25 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:772:5 (python+0x72634e)
    #26 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #27 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1)
    #28 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145)
    #29 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)
    #30 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)

SUMMARY: ThreadSanitizer: data race /home/shamil/oss/cpython/main/Python/pystate.c:769:26 in interpreter_clear
==================
[4156 refs, 1966 blocks]

Analysis

The race occurs because:

Main thread starts finalization and calls interpreter_clear() which writes to _PyRuntime.ceval.eval_breaker
Meanwhile, daemon threads are still running and attempt to acquire the GIL via take_gil()
take_gil() calls _Py_unset_eval_breaker_bit() which performs atomic operations on the same eval_breaker field
These accesses are not properly synchronized during the finalization phase

Environment

CPython version: main branch (commit: [fbf0843])
Operating System: Linux | Ubuntu 25.10
Compiler: Ubuntu clang version 20.1.8 (0ubuntu4)
Build flags: --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes

Related Issues

race condition in threading when interpreter finalized while daemon thread runs (thread sanitizer identified) #124878 - Similar race during finalization but with drop_gil and free_threadstate
test__xxsubinterpreters is Occasionally Crashing #104341 - Background on interpreter deletion during finalization

CPython versions tested on:

3.15, CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

ashm-dev
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
on Oct 17, 2025
colesbury commented on Oct 17, 2025
colesbury
on Oct 17, 2025
Contributor
More actions
We should just change the assignment at:

cpython/Python/pystate.c

Line 769 in fbf0843

tstate->eval_breaker = 0;

To an atomic operation and update the code comment.
ashm-dev commented on Oct 17, 2025
ashm-dev
on Oct 17, 2025
ContributorAuthor
More actions
@colesbury Can I start fixing this bug?
colesbury commented on Oct 17, 2025
colesbury
on Oct 17, 2025
Contributor
More actions
Yes, please
Reacted by Shamil
StanFromIreland
added
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)
on Oct 17, 2025
kumaraditya303
added a commit that references this issue on Oct 18, 2025
gh-140257: fix data race on eval_breaker during finalization (#140265)
c8729c9
miss-islington
added a commit that references this issue on Oct 18, 2025
pythongh-140257: fix data race on eval_breaker during finalization (p…
f864c6c
kumaraditya303
linked a pull request that will close this issue[3.14] gh-140257: fix data race on eval_breaker during finalization (GH-140265) #140294 on Oct 18, 2025
kumaraditya303
added a commit that references this issue on Oct 18, 2025
[3.14] gh-140257: fix data race on eval_breaker during finalization (G…
3ca7ea1
kumaraditya303
closed this as completed on Oct 18, 2025
StanFromIreland
added a commit that references this issue on Dec 6, 2025
pythongh-140257: fix data race on eval_breaker during finalization (p…
4b1a040

Metadata

Assignees

No one assigned

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

ThreadSanitizer: data race in interpreter_clear() vs take_gil() during finalization with daemon threads #140257

Description

Bug report

Bug description:

How to Reproduce

TSAN Output

Analysis

Environment

Related Issues

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

colesbury commented on Oct 17, 2025

ashm-dev commented on Oct 17, 2025

colesbury commented on Oct 17, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions