GeezerVison is showing github.com — Segfault when cleaning up after a MemoryError with deeply nested objects · Issue #149146 · python/cpython
Segfault when cleaning up after a MemoryError with deeply nested objects #149146DescriptionCrash report
What happened?
When in memory-constrained situations, the following code generates a segfault on main:
b = 9
while True:
b = (b, None)Clearly, b grows to be an unbounded nested tuple, which eventually exhausts the memory.
A MemoryError is correctly thrown, but during unwind, python tries to clean up the object, and this triggers a stack overflow.
Example stack
Process 369 stopped
Thread 1 "fuzz_python" stop reason = signal SIGSEGV
frame #0: frame #0: 0x0000aaaaaea31648 fuzz_python`tuple_dealloc(self=0x0000ffff9fa045e0) at tupleobject.c:256
frame #1: frame #1: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa045e0) at object.c:3303:5
frame #2: frame #2: 0x0000aaaaaea3182c fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #3: frame #3: 0x0000aaaaaea3180c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
frame #4: frame #4: 0x0000aaaaaea31804 fuzz_python`tuple_dealloc(self=0x0000ffff9fa04630) at tupleobject.c:277:9
frame #5: frame #5: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa04630) at object.c:3303:5
frame #6: frame #6: 0x0000aaaaaea3182c fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #7: frame #7: 0x0000aaaaaea3180c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
...
frame #2709: frame #2709: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa11cd0) at object.c:3303:5
frame #2710: frame #2710: 0x0000aaaaaea3182c fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #2711: frame #2711: 0x0000aaaaaea3180c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
frame #2712: frame #2712: 0x0000aaaaaea31804 fuzz_python`tuple_dealloc(self=0x0000ffff9fa11d20) at tupleobject.c:277:9
frame #2713: frame #2713: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa11d20) at object.c:3303:5
frame #2714: frame #2714: 0x0000aaaaaea3182c fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #2715: frame #2715: 0x0000aaaaaea3180c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
frame #2716: frame #2716: 0x0000aaaaaea31804 fuzz_python`tuple_dealloc(self=0x0000ffff9fa11d70) at tupleobject.c:277:9
frame #2717: frame #2717: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa11d70) at object.c:3303:5
frame #2718: frame #2718: 0x0000aaaaaea3182c fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #2719: frame #2719: 0x0000aaaaaea3180c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
frame #2720: frame #2720: 0x0000aaaaaea31804 fuzz_python`tuple_dealloc(self=0x0000ffff9fa11dc0) at tupleobject.c:277:9
frame #2721: frame #2721: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffff9fa11dc0) at object.c:3303:5
frame #2722: frame #2722: 0x0000aaaaae98f290 fuzz_python`Py_DECREF(op=<unavailable>) at refcount.h:427:9 [inlined]
frame #2723: frame #2723: 0x0000aaaaae98f26c fuzz_python`Py_XDECREF(op=<unavailable>) at refcount.h:520:9 [inlined]
frame #2724: frame #2724: 0x0000aaaaae98f26c fuzz_python`dictkeys_decref(dk=0x0000ffffbe045210, use_qsbr=false) at dictobject.c:495:17
frame #2725: frame #2725: 0x0000aaaaae98630c fuzz_python`dict_dealloc(self=0x0000ffffbe055f40) at dictobject.c:0
frame #2726: frame #2726: 0x0000aaaaae9ccd54 fuzz_python`_Py_Dealloc(op=0x0000ffffbe055f40) at object.c:3303:5
frame #2727: frame #2727: 0x0000aaaaae7ecb8c fuzz_python`Py_DECREF(op=0x0000ffffbe055f40) at refcount.h:427:9 [inlined]
frame #2728: frame #2728: 0x0000aaaaae7ecb58 fuzz_python`main(argc=1, argv=0x0000fffff67f4238) at fuzz_python.c:226:17
frame #2729: frame #2729: 0x0000ffffbe1e1f9c ld-musl-aarch64.so.1`libc_start_main_stage2(main=(fuzz_python`main at fuzz_python.c:71), argc=1, argv=0x0000fffff67f4238) at __libc_start_main.c:95:2
The following dockerfile reliably reproduces it for me:
Dockerfile
FROM alpine:latest
RUN apk add --no-cache \
git build-base linux-headers openssl-dev zlib-dev bzip2-dev \
xz-dev sqlite-dev readline-dev libffi-dev ncurses-dev gdbm-dev \
tcl-dev tk-dev
WORKDIR /src
RUN git clone --depth=1 https://github.com/python/cpython.git
WORKDIR /src/cpython
RUN ./configure && make -j2
RUN cat > /tc.py <<'PY'
b = 9
while True:
b = (b, None)
PY
RUN cat > /entrypoint.sh <<'SH'
#!/bin/sh
echo "Running test"
# 512 MiB virtual memory limit
ulimit -v 524288
/src/cpython/python /tc.py
echo "Python process exited with code $?"
SH
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Gives me:
>docker run --rm -it 29apr
Running test
Traceback (most recent call last):
File "/tc.py", line 3, in <module>
object address : 0xffffae809fc0
object refcount : 3
object type : 0xaaaad32175a8
object type name: MemoryError
object repr : MemoryError()
lost sys.stderr
Segmentation fault
Python process exited with code 139
ISTR that stack overflows from recursive objects were at least considered low priority, if not 'not worth fixing' before, but I can't recall where I read that, it was a while ago, so apologies if this isn't a useful report!
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.15.0a8+ (heads/main:d4eee16, Apr 29 2026, 11:03:26) [GCC 15.2.0]
Linked PRs
Activitystestagg commented on Apr 29, 2026 ContributorAuthor More actions Bisect identified this as the change that introduced the segfault: @markshannon
147cd05 is the first bad commit
commit 147cd05
Author: Mark Shannon
Date: Wed Apr 17 11:08:05 2024 +0100
GH-117760: Streamline the trashcan mechanism (GH-117763)
Include/cpython/object.h | 47 ++++++++--------
Include/cpython/pystate.h | 7 +--
Include/internal/pycore_object.h | 1 -
Objects/object.c | 114 ++++++---------------------------------
Python/pystate.c | 2 +
5 files changed, 43 insertions(+), 128 deletions(-)
bisect found first bad commit bhuvi27 commented on Jun 16, 2026 Hi — I'd like to take a look at this. Reproducing locally and reading through the tuple_dealloc path from the bisect. Will open a PR if I get somewhere. vstinner commented on Jun 18, 2026 The trace includes "musl": the musl C library. I recently modified Python to update runtime stack size checks on musl. See commit 9a61d1c (issue gh-151546).
Can you please check if you can still reproduce the issue on the current main branch? stestagg commented on Jun 18, 2026 ContributorAuthor More actions @vstinner - Thank you! I've actually switched away from musl since this report, due to some memory issues, so that's good news.
My quick tests indicates this is still reproducible wthout musl:
echo 2 > /proc/sys/vm/overcommit_memory
~ #
~ # /pfm/py/bin/python3 /pfm/scratch/dist/gh149146.py
[ 487.622390] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622514] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622595] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622646] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622759] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622840] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622878] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622936] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.622987] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
[ 487.623022] __vm_enough_memory: pid: 97, comm: python3, bytes: 1048576 not enough memory for the allocation
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 1340, in _find_and_load
File "<frozen importlib._bootstrap>", line 1290, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1216, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1270, in find_spec
File "<frozen importlib._bootstrap_external>", line 1244, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1364, in find_spec
File "<frozen importlib._bootstrap_external>", line 1407, in _fill_cache
OSError: [Errno 12] Cannot allocate memory: '/pfm/scratch/dist'
Traceback (most recent call last):
File "/pfm/scratch/dist/gh149146.py", line 3, in <module>
object address : 0xfffff7c27700
object refcount : 3
object type : 0xaaaaab78d7b0
object type name: MemoryError
object repr : MemoryError()
lost sys.stderr
Segmentation fault
~ # /pfm/py/bin/python3 --version
Python 3.16.0a0
~ # ldd /pfm/py/bin/python3
linux-vdso.so.1 (0x0000fffff7fff000)
libm.so.6 => /usr/lib/libm.so.6 (0x0000fffff6e30000)
libc.so.6 => /usr/lib/libc.so.6 (0x0000fffff6c60000)
/lib/ld-linux-aarch64.so.1 => /usr/lib/ld-linux-aarch64.so.1 (0x0000fffff7fb0000)
that was with:
commit a064b323f4350305e7486c7b1090cf12b19e7738 (grafted, HEAD -> main, origin/main)
Author: Victor Stinner <[email protected]>
Date: Wed Jun 17 10:58:22 2026 +0200
gh-151496: Use process groups in test_dtrace (#151512)
...
I'll run again with latest main to verify stestagg commented on Jun 18, 2026 ContributorAuthor More actions Yea, it looks to still be there:
~ # echo 2 > /proc/sys/vm/overcommit_memory
~ # lldb /pfm/py/bin/python3 /pfm/scratch/gh149146.py
[ 28.116418] random: crng init done
(lldb) target create "/pfm/py/bin/python3"
Current executable set to '/pfm/py/bin/python3' (aarch64).
(lldb) settings set -- target.run-args "/pfm/scratch/gh149146.py"
(lldb) run
Process 90 launched: '/pfm/py/bin/python3' (aarch64)
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 1340, in _find_and_load
File "<frozen importlib._bootstrap>", line 1290, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1216, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1270, in find_spec
File "<frozen importlib._bootstrap_external>", line 1244, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1364, in find_spec
File "<frozen importlib._bootstrap_external>", line 1407, in _fill_cache
OSError: [Errno 12] Cannot allocate memory: '/pfm/scratch'
Traceback (most recent call last):
File "/pfm/scratch/gh149146.py", line 3, in <module>
object address : 0xfffff7c27700
object refcount : 3
object type : 0xaaaaab78d870
object type name: MemoryError
object repr : MemoryError()
lost sys.stderr
Process 90 stopped
* thread #1, name = 'python3', stop reason = signal SIGSEGV: address not mapped to object (fault address=0xfffffffddff0)
frame #0: 0x0000aaaaaafebc58 python3`_Py_Dealloc(op=0x0000aaaaabdeac90) at object.c:3291
3288 stack is shallower */
3289 void
3290 _Py_Dealloc(PyObject *op)
-> 3291 {
3292 PyTypeObject *type = Py_TYPE(op);
3293 unsigned long gc_flag = type->tp_flags & Py_TPFLAGS_HAVE_GC;
3294 destructor dealloc = type->tp_dealloc;
~ # /pfm/py/bin/python3 -VV
Python 3.16.0a0 (heads/main-dirty:12add38, Jun 18 2026, 11:18:38) [Clang 22.1.6 ]
vstinner commented on Jun 18, 2026 Ok, I reproduced the issue on Fedora 44 (using glibc). I limited the virtual memory to ~1 GiB:
$ ulimit -v 1024000
$ gdb -args ./python b.py
(gdb) run
gdb traceback:
#0 0x00000000005717f6 in _Py_RecursionLimit_GetMargin ()
#1 0x0000000000577dd0 in _Py_Dealloc ()
#2 0x00000000005ae871 in Py_DECREF ()
#3 0x00000000005ae8a1 in Py_XDECREF ()
#4 0x00000000005aff6c in tuple_dealloc ()
#5 0x0000000000577ed5 in _Py_Dealloc ()
#6 0x00000000005ae871 in Py_DECREF ()
#7 0x00000000005ae8a1 in Py_XDECREF ()
#8 0x00000000005aff6c in tuple_dealloc ()
#9 0x0000000000577ed5 in _Py_Dealloc ()
#10 0x00000000005ae871 in Py_DECREF ()
#11 0x00000000005ae8a1 in Py_XDECREF ()
#12 0x00000000005aff6c in tuple_dealloc ()
#13 0x0000000000577ed5 in _Py_Dealloc ()
#14 0x00000000005ae871 in Py_DECREF ()
#15 0x00000000005ae8a1 in Py_XDECREF ()
#16 0x00000000005aff6c in tuple_dealloc ()
#17 0x0000000000577ed5 in _Py_Dealloc ()
#18 0x00000000005ae871 in Py_DECREF ()
#19 0x00000000005ae8a1 in Py_XDECREF ()
#20 0x00000000005aff6c in tuple_dealloc ()
#21 0x0000000000577ed5 in _Py_Dealloc ()
#22 0x00000000005ae871 in Py_DECREF ()
...
Stack of the process:
$ grep stack /proc/1285237/maps
7ffffffc8000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
That's only 220 kiB, it's quite small! Glibc uses a stack size of 8 MiB and the stack is allocated on demand. Here, the stack cannot be allocated since we exhausted all the memory.
There isn't much that Python can do in this case: when the memory is exhausted, Python fails to allocate more stack and so it does crash with a stack overflow.
Another reproducer which limits the process memory to 1 GiB to not crash the whole system:
import resource
# deallocation the chain uses a lot of C stack memory
CHAIN_LENGTH = 1024 * 1024
# limit process memory to not crash the whole machine
max_memory = 1024 * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (max_memory, max_memory))
print("create long chain...")
long_chain = None
for _ in range(CHAIN_LENGTH):
long_chain = (long_chain, None)
print("fill the process memory...")
memory = []
try:
large = 1024 * 1024
while True:
memory.append(b'x' * large)
except MemoryError:
pass
small = 1024
while True:
memory.append(b'x' * small)
# at exit (on error), deallocating long_chain will use a lot of C stack memory
# ... and does crash!stestagg commented on Jun 18, 2026 ContributorAuthor More actions Aah, I hadn't realised that this was that stack resize failure issue (thanks!) I'd already separately raised #150722 to cover that, and agree it's a hard one to detect.
@bhuvi27 has a PR, but I have no idea if that approach is sensible or not. vstinner commented on Jun 24, 2026 Oh, when I ran my reproducer script, I found yet another bug related to error handling of MemoryError: issue #152090. vstinner commented on Jun 24, 2026 So the problem is that the system/process memory is exhausted, and so when the Linux kernel tries to extend the process stack memory, it fails with a segfault since there is no more available memory.
There is nothing Python can do in this case. There is no generic and safe mechanism to catch the segfault and report it as a regular Python exception. Catching SIGSEGV signal is dangerous. vstinner commented on Jul 10, 2026 I close the issue as "won't fix".
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|