GeezerVison is showing github.com — `deque_clear` clears `MemoryError` · Issue #152017 · python/cpython
deque_clear clears MemoryError #152017
DescriptionBug report
Bug description:
This is a sub-issue for gh-146102 with the gist here.
In the case that there is no memory, deque_clear calls PyErr_Clear():
|
b = newblock(deque); |
|
if (b == NULL) { |
|
PyErr_Clear(); |
|
goto alternate_method; |
|
} |
This was done to clear the error that newblock raises. However, if there was originally an exception, deque_clear clears it.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activitychanged the title [-]'deque_clear' clears `MemoryError`[/-] [+]`deque_clear` clears `MemoryError`[/+] on Jun 23, 2026rhettinger commented on Jun 26, 2026 What is wrong with the current code? If an allocation fails due to lack of memory (a very rare almost non-existent event), then an alternate path is taken that does not require additional memory.
AFAICT this is a false positive from the umbrella issue. brijkapadia commented on Jun 27, 2026 ContributorAuthor More actions
AFAICT this is a false positive from the umbrella issue.
There is a real error here that the reproducer catches. However, the umbrella issue gives the wrong explanation. The problem has nothing to do with deque.copy but deque.copy is one execution path for the error.
Reproducer (for reference)
import _testcapi
from collections import deque
for n in range(1, 80):
d = deque(range(200))
_testcapi.set_nomemory(n, 0)
try:
d2 = d.copy()
_testcapi.remove_mem_hooks()
break
except MemoryError:
_testcapi.remove_mem_hooks()
except:
_testcapi.remove_mem_hooks()
# Fatal: _Py_Dealloc: Deallocator of type 'collections.deque' clearedThe real problem is that this debug check in _Py_Dealloc fails:
Debug Check
|
#ifdef Py_DEBUG |
|
// gh-89373: The tp_dealloc function must leave the current exception |
|
// unchanged. |
|
if (tstate != NULL && tstate->current_exception != old_exc) { |
|
const char *err; |
|
if (old_exc == NULL) { |
|
err = "Deallocator of type '%s' raised an exception"; |
|
} |
|
else if (tstate->current_exception == NULL) { |
|
err = "Deallocator of type '%s' cleared the current exception"; |
|
} |
|
else { |
|
// It can happen if dealloc() normalized the current exception. |
|
// A deallocator function must not change the current exception, |
|
// not even normalize it. |
|
err = "Deallocator of type '%s' overrode the current exception"; |
|
} |
|
_Py_FatalErrorFormat(__func__, err, type->tp_name); |
|
} |
|
Py_XDECREF(old_exc); |
|
Py_DECREF(type); |
|
#endif |
For the specific case of the reproducer, deque_copy_impl calls deque_append_lock_held and in the case of an error, calls Py_DECREF(new_deque):
|
if (deque_append_lock_held(new_deque, Py_NewRef(item), |
|
new_deque->maxlen) < 0) { |
|
Py_DECREF(new_deque); |
|
return NULL; |
|
} |
Furthermore, deque_append_lock_held calls newblock which raises a PyErr_NoMemory() in the case of OOM:
Thus, in the case of OOM, Py_DECREF(new_deque) is called with an exception already set.
In order to free the memory, eventually deque_clear is called. deque_clear calls newblock, which in the case of OOM, replaces the current exception with a new PyErr_NoMemory(). Then, deque_clear calls PyErr_Clear(), meaning that there are now no exceptions despite deque_append_lock_held originally setting one:
|
b = newblock(deque); |
|
if (b == NULL) { |
|
PyErr_Clear(); |
|
goto alternate_method; |
|
} |
In debug builds, this is caught in _Py_Dealloc,
|
else if (tstate->current_exception == NULL) { |
|
err = "Deallocator of type '%s' cleared the current exception"; |
|
} |
which gives the error Fatal: _Py_Dealloc: Deallocator of type 'collections.deque' cleared the current exception. stestagg commented on Jul 2, 2026 EDIT: ignore my previous, it was something else.
I just came across this in fuzzing, the following code reliably aborts for me:
import collections
d =33333333333* collections.deque([10]) gives (in linux with 2gb ram)
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y
~ # /pfm/tools/mem_limit_exec /pfm/py/bin/python3 /pfm/scratch/reproducers/t1-1.
min.py
Fatal Python error: _Py_Dealloc: Deallocator of type 'collections.deque' cleared the current exception
Python runtime state: initialized
Current thread 0x0000fffff7fee020 [python3] (most recent call first):
File "<input-53>", line 2 in <module>
File "/pfm/scratch/reproducers/t1-1.min.py", line 10 in <module>
Aborted
~ # /pfm/tools/mem_limit_exec /pfm/py/bin/python3 /pfm/scratch/reproducers/t1-1.
min.py
Fatal Python error: _Py_Dealloc: Deallocator of type 'collections.deque' cleared the current exception
Python runtime state: initialized
Current thread 0x0000fffff7fee020 [python3] (most recent call first):
File "<input-53>", line 2 in <module>
File "/pfm/scratch/reproducers/t1-1.min.py", line 10 in <module>
Aborted
~ #
Python 3.16.0a0 (heads/main-dirty:f8514dc, Jul 2 2026, 09:14:40) [Clang 22.1.6 ]
rhettinger commented on Oct 26, 2026 Do you have a plain Python reproducer that doesn't rely on test_capi? Does this ever happen in real code? brijkapadia commented on Oct 28, 2026 ContributorAuthor More actions
Do you have a plain Python reproducer that doesn't rely on test_capi? Does this ever happen in real code?
No. There is no doubt that this is an edge case that usually does not occur. However, I think #152018 gives a relatively simple solution to this problem so its probably worth resolving (IMO). stestagg commented on Oct 28, 2026 As per my comment above, this is easy to reproduce in memory-limited situations without needing test_capi, using this code:
granted, it's a memory exhaustion situation, but this error is triggered.
import collections
d = 33333333333 * collections.deque([10])
As an example, the following script reliably works for me on Linux (macos rlimit stuff, IIRC doesn't work)
import collections, platform, resource, sys
print(platform.platform(), sys.version)
resource.setrlimit(resource.RLIMIT_AS, (2**31, 2**31))
33333333333 * collections.deque([10])
[sstagg@xps scratch]$ python test_dq_152017.py
Linux-7.2.7-arch1-1-x86_64-with-glibc2.44 3.14.2 (main, Feb 3 2026, 12:51:13) [GCC 15.2.1 20251112]
Traceback (most recent call last):
File "/home/sstagg/src/scratch/test_dq_152017.py", line 4, in <module>
33333333333 * collections.deque([10])
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
SystemError: error return without exception set
AND in a debug build:
[sstagg@xps cpython]$ ./python ~/src/scratch/test_dq_152017.py
Linux-7.2.7-arch1-1-x86_64-with-glibc2.44 3.16.0a1+dev (heads/main:e7ab898188c, Oct 28 2026, 08:35:51) [GCC 16.2.1 20260810]
Fatal Python error: py_dealloc: Deallocator of type 'collections.deque' cleared the current exception
Python runtime state: initialized
Current thread 0x00007f02d7b16780 [python] (most recent call first):
File "/home/sstagg/src/scratch/test_dq_152017.py", line 4 in <module>
Aborted (core dumped) ./python ~/src/scratch/test_dq_152017.py
stestagg commented on Oct 28, 2026 For a real-world impact that isn't just a 'ran out of memory, program terminated' issue, here's a reproducer that, for me, silently gives invalid outputs if this error is triggered:
import os
import resource
from collections import deque
from itertools import accumulate
N = 2_000_000
chunks = (deque([None]) * N, deque([None]) * N)
expected_length = sum(map(len, chunks)) # 4,000,000
## Set the resource limit to force a MemoryError reliaby..
page_size = os.sysconf("SC_PAGE_SIZE")
with open("/proc/self/statm", encoding="ascii") as f:
current_vsize = int(f.read().split()[0]) * page_size
limit = current_vsize + 8 * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))
###
last_result = None
results_produced = 0
for last_result in accumulate(chunks):
results_produced += 1
actual_length = len(last_result)
print(f"Expected final length: {expected_length:,}")
print(f"Actual final length: {actual_length:,}")
print(f"Expected results: {len(chunks)}")
print(f"Actual results: {results_produced}")
sstagg@xps scratch]$ python test_dq2_152017.py
Expected final length: 4,000,000
Actual final length: 2,000,000
Expected results: 2
Actual results: 1
This script was made for me by daybreak blue
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|