GeezerVison is showing github.com — sub-interpreters: significant memory leak on shutdown · Issue #110411 · python/cpython
sub-interpreters: significant memory leak on shutdown #110411Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
Hi all, congrats on the new Python release, very keen on the direction that sub-interpreters are taking!
I noticed that constructing and destructing sub-interpreters now causes memory to be leaked. This was not a problem in 3.11, but is as of 3.12 and occurs on latest main branch too (as of 8c07137). I figure it must relate to the changes made to split up the interpreter states.
I can consistently replicate the issue using simply:
import _xxsubinterpreters
while 1:
interp = _xxsubinterpreters.create()
_xxsubinterpreters.destroy(interp)You'll notice the memory usage climb quite quickly if you look at top/htop.
valgrind/massif think the culprit are interned strings, specifically here:
==219777== at 0x4848A13: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==219777== by 0x2DA59E: PyMem_RawCalloc (obmalloc.c:671)
==219777== by 0x2DA59E: arena_map_get (obmalloc.c:1021)
==219777== by 0x2DA59E: arena_map_mark_used (obmalloc.c:1076)
==219777== by 0x2DAA71: new_arena (obmalloc.c:1209)
==219777== by 0x2DAA71: allocate_from_new_pool (obmalloc.c:1389)
==219777== by 0x2DB00B: pymalloc_alloc (obmalloc.c:1553)
==219777== by 0x2DB00B: _PyObject_Malloc (obmalloc.c:1564)
==219777== by 0x424AF5: gc_alloc (gcmodule.c:2304)
==219777== by 0x424AF5: _PyObject_GC_New (gcmodule.c:2319)
==219777== by 0x2BDF3B: new_dict (dictobject.c:748)
==219777== by 0x2BDF3B: PyDict_New (dictobject.c:851)
==219777== by 0x34CF06: init_interned_dict (unicodeobject.c:248)
==219777== by 0x34CF06: _PyUnicode_InitGlobalObjects (unicodeobject.c:14670)
==219777== by 0x3EDE18: pycore_init_global_objects (pylifecycle.c:678)
==219777== by 0x3EDE18: pycore_interp_init (pylifecycle.c:826)
==219777== by 0x3F07A9: new_interpreter (pylifecycle.c:2079)
==219777== by 0x3F07A9: Py_NewInterpreterFromConfig (pylifecycle.c:2114)
==219777== by 0x4867A52: interp_create (_xxsubinterpretersmodule.c:516)
==219777== by 0x2D0C22: cfunction_call (methodobject.c:537)
==219777== by 0x2716A7: _PyObject_MakeTpCall (call.c:240)
I had a look around that code but I can't spot any obvious bug. I did verify that the interned strings dict created and destroyed in clear_interned_dict() is the same for each sub-interpreter, just in case.
I think every interned string is now also automatically immortal. Does that mean it wouldn't be cleaned up even if the refcount goes down to zero?
CPython versions tested on:
3.11, 3.12, CPython main branch
Operating systems tested on:
Linux Reacted by dawmster, Claus Holbech and Uwe Höhna Activitycostasgambit commented on Oct 6, 2023 I can reproduce this from C as well, without using _xxsubinterpreters. itamaro commented on Oct 6, 2023 ehenson commented on Nov 13, 2023 I can reproduce as well. I'm using Go with CGO.
This is within a Gin route Post handler.
var config = &C.PyInterpreterConfig{
use_main_obmalloc: 0,
allow_fork: 0,
allow_exec: 0,
allow_threads: 1,
allow_daemon_threads: 0,
check_multi_interp_extensions: 1,
gil: C.PyInterpreterConfig_OWN_GIL,
}
runtime.LockOSThread()
// Create a new subinterpreter
var tstate *C.PyThreadState
status := C.Py_NewInterpreterFromConfig(&tstate, config)
// compile and execute a tiny Python script as if it were an extension point for my application.
// paying very close attention to reference counting.
handleRequest(c)
C.Py_EndInterpreter(tstate)
runtime.UnlockOSThread()
})
When using a stress tester, the memory grows into gigabytes within a few seconds, consistently growing, and never recovers. dawmster commented on Dec 11, 2023 ehenson commented on Dec 17, 2023 I am now using 3.12.1, and it is still leaking. I am using MacOS 14.2 M2 and Linux amd64.
I have studied the documentation about using Py_NewInterpreterFromConfig, and I doubt that I am using it correctly. If someone can post here a C example of using Py_NewInterpreterFromConfig, the Python script is a simple print("Hello, World!"), the C example is in the proper form of using Py_NewInterpreterFromConfig concerning managing the GIL and every step needed until Py_EndInterpreter; I will confirm my code and gladly report my results. ericsnowcurrently commented on Jan 25, 2024 Sorry, this slipped through the cracks for me. 😞
I expect the underlying cause is the same as for gh-113055 and kind of has the same fix (gh-113412). Basically, the obmalloc implementation has never freed the blocks we get from the system allocator. There's always been a leak! It was just much harder to notice before. The difference between 3.11 and 3.12+ is that each interpreter has its own obmalloc state rather than sharing a process-global one like we used to. Consequently, all the memory each (isolated) subinterpreter uses leaks now.
Let's circle back on this once gh-113412 is merged and maybe also gh-113601. picnixz commented on Dec 2, 2024 hackhaslam commented on Aug 18, 2025 This is still a problem in 3.13.7 and 3.14.0rc2. Is this on anyone's radar?
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error StatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|