GeezerVison is showing github.com — `collections.deque` is not atomic during copy · Issue #144809 · python/cpython
collections.deque is not atomic during copy #144809
Description[Edit: This bug was originally reported as "not thread safe", and updated to "not atomic"]
Bug report
Bug description:
In free-threaded mode, calling copy on collections.deque is not thread safe atomic and can raise RuntimeError: deque mutated during iteration errors.
Example code:
import sys
import threading
from collections import deque
from copy import copy
print(f"Python {sys.version}")
print(f"GIL enabled: {sys._is_gil_enabled()}")
d = deque(range(100))
stop = threading.Event()
def mutate():
i = 0
while not stop.is_set():
d.append(i)
if len(d) > 200:
d.popleft()
i += 1
def copy_loop():
while not stop.is_set():
copy(d)
barrier = threading.Barrier(5)
def run(fn):
barrier.wait()
fn()
threads = [
threading.Thread(target=run, args=(f,), daemon=True)
for f in [mutate, mutate, copy_loop, copy_loop]
]
for t in threads:
t.start()
barrier.wait()
stop.wait(timeout=5)
stop.set()
for t in threads:
t.join(timeout=5)
print("Done")Running under free-threaded build:
❯ ./python /tmp/deque.py
Python 3.15.0a6+ free-threading build (heads/main:945bf8ce1bf, Feb 13 2026, 22:04:08) [GCC 11.4.0]
GIL enabled: False
Exception in thread Thread-4 (run):
Exception in thread Thread-3 (run):
Traceback (most recent call last):
Traceback (most recent call last):
File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "cpython/Lib/threading.py", line 1017, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/deque.py", line 32, in run
fn()
~~^^
File "/tmp/deque.py", line 24, in copy_loop
copy(d)
~~~~^^^
File "cpython/Lib/copy.py", line 82, in copy
return copier(x)
RuntimeError: deque mutated during iteration
File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "cpython/Lib/threading.py", line 1017, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/deque.py", line 32, in run
fn()
~~^^
File "/tmp/deque.py", line 24, in copy_loop
copy(d)
~~~~^^^
File "cpython/Lib/copy.py", line 82, in copy
return copier(x)
RuntimeError: deque mutated during iteration
DoneCPython versions tested on:
CPython main branch, 3.14
Operating systems tested on:
Linux
Linked PRs
Activityserhiy-storchaka commented on Feb 15, 2026 I do not think it is not thread safe. Copying does not crash Python, does not corrupt memory, and does not left a deque in an inconsistent state. It is not atomic -- but you cannot in general expect that an operation with non-constant complexity on collection is atomic.
cc @rhettinger yilei commented on Feb 16, 2026 ContributorAuthor More actions Thanks for taking a look at this!
Hmm, my mental model of thread safety is that I'd expect copy(d) would reasonably give me a copy when there are other concurrent mutations in parallel, without using e.g. a lock to guard the copy and mutations. And perhaps that's a wrong interpretation? And non-atomic is the better term. My assumption was largely based on:
- Seeing earlier free-threading work from gh-112050: Make collections.deque thread-safe in free-threaded builds #113830
- Similar code for
dict and list succeeds without errors
Btw, additional context for this bug report is getsentry/sentry-python#5465 where we encountered this issue.
I'm fine if we couldn't make copy(d) work here without users adding guards around calls to deque methods (though IMO it hurts usability), it would be great if we could document this expectation somewhere. colesbury commented on Feb 17, 2026 I think we should make this behave atomically because:
- it's written so that it behaves atomically in the GIL-enabled build.
- it should be relatively easy (... famous last words)
- the motivation is a real bug report in user code
The problem is a combination of:
- We lock the to-be-copied deque when
__copy__ is called (good!)
- We iterate over the to-be-copied deque, which tries to lock both the iterator and the deque
That triggers the deadlock avoidance routine, which unlocks the deque before re-locking both the iterator and the deque. The time in between the unlock and re-locking allows another thread to modify the deque (which would not be allowed in the GIL-enabled build). Yhg1s commented on Feb 18, 2026 Given that we already have an optimization where we skip locking (directly) nested critical sections of the same object, and we order CRITICAL_SECTION2 locks by mutex address to avoid deadlocks (so the order of it/deque locking is already opaque to the caller) -- couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section? yilei commented on Feb 18, 2026 ContributorAuthor More actions
couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?
I think this works, and it's a good optimization to have anyway. I'll clean up my draft code, add some tests, and send a PR. colesbury commented on Feb 18, 2026
couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?
That would work for the specific code in deque, but it's not safe in general because if the thread state is detached and then attached, only the (new) top-most critical section is locked. For example:
Py_BEGIN_CRITICAL_SECTION(a);
Py_BEGIN_CRITICAL_SECTION2(a, b); // only locks b?
_PyThreadState_Detach(tstate);
_PyThreadState_Attach(tstate);
// it's important that both a and b are locked here.
// if the Py_BEGIN_CRITICAL_SECTION2 only pushed
// a critical section containing b, then only b will be locked
Py_END_CRITICAL_SECTION2();
Py_END_CRITICAL_SECTION(); I think we could probably get something to work safely along the lines of "only lock b", but I think it'll be complicated. colesbury commented on Feb 18, 2026 #144966 is a small change that I'd be comfortable backporting to 3.14.
It'd be great to add an optimization for CRITICAL_SECTION2, but I think that will be a lot more complicated. yilei commented on Feb 18, 2026 ContributorAuthor More actions changed the title [-]`collections.deque` is not thread safe during copy[/-] [+]`collections.deque` is not atomic during copy[/+] on Feb 18, 2026
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|