GeezerVison is showing github.com — Memory leak on executables embedded with 3.13 · Issue #113055 · python/cpython
Memory leak on executables embedded with 3.13 #113055DescriptionBug report
Bug description:
Since 67807cf, _testembed.exe (x64 Release) increases the memory usage in the cycle of Py_Initialize and Py_Finalize.
I monitored the Task Manager (taskmgr.exe) on Windows10/11, with a change against Programs/_testembed.c:
- #define INIT_LOOPS 4
+ #define INIT_LOOPS 200 Commands:
_testembed.exe test_repeated_simple_init orpython -m test test_embed -m test_simple_initialization_api (no log)
With RADIX TREE (as-is)
| Loop |
67807cf |
main |
3.11.7 |
| 100 |
67MB |
210MB |
4MB |
| 200 |
135MB |
406MB |
4MB |
No RADIX TREE (as-is)
| Loop |
67807cf |
main |
3.11.7 |
| 100 |
10MB |
168MB |
4MB |
| 200 |
16MB |
336MB |
4MB |
Recent obmalloc.c looks ready for finalization. Just adding my rough (invalid?) experiment made the leaks even. So, I hope that they share the same issue. Otherwise, ea2c001 or 15d4c9f has another leak. #98359 (comment)
patch
void
_PyInterpreterState_FinalizeAllocatedBlocks(PyInterpreterState *interp)
{
if (has_own_state(interp)) {
Py_ssize_t leaked = _PyInterpreterState_GetAllocatedBlocks(interp);
assert(has_own_state(interp) || leaked == 0);
interp->runtime->obmalloc.interpreter_leaks += leaked;
+ OMState *state = &interp->obmalloc;
+ for (uint i = 0; i < maxarenas; ++i) {
+ if (allarenas[i].address == 0) {
+ continue;
+ }
+ _PyObject_Arena.free(_PyObject_Arena.ctx,
+ (void *)allarenas[i].address, ARENA_SIZE);
+ allarenas[i].address = 0;
+ --narenas_currently_allocated;
+ }
+ PyMem_RawFree(allarenas);
}
}
With RADIX TREE (patched)
| Loop |
67807cf |
main |
| 100 |
42MB |
40MB |
| 200 |
80MB |
80MB |
No RADIX TREE (patched)
| Loop |
67807cf |
main |
| 100 |
3.5M |
3.3M |
| 200 |
3.7M |
3.4M |
cc @ericsnowcurrently @nascheme
CPython versions tested on:
3.12, 3.13, CPython main branch: 3aea6c4
Operating systems tested on:
Windows
Linked PRs
Reacted by Eric Snow, sunmy2019 and Osyotr Activitynascheme commented on Dec 13, 2023 That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free. neonene commented on Dec 13, 2023 ContributorAuthor More actions
How are you measuring memory usage and what's the OS?
Only the Task Manager on Windows 10 and 11 for me.
The microsoft.com offers VMMap as a small alternative. changed the title [-]Memory leak on embedded Windows executable with 3.12 and later[/-] [+]Memory leak on Windows executable embedded with 3.12 and later[/+] on Dec 15, 2023nascheme commented on Dec 16, 2023 @ericsnowcurrently
I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to 67807cfc it didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.
A possible way to fix this: when free_interpreter() gets called, walk through obmalloc arenas and free them. The pymalloc_print_stats() function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.
If you free the arenas and there is allocated blocks inside of them, you can't call pymalloc_free() on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters. nascheme commented on Dec 16, 2023 Building gh-113218 with the address sanitizer and then running ./Programs _testembed test_repeated_simple_init shows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this. Reacted by neonene and Eric Snow changed the title [-]Memory leak on Windows executable embedded with 3.12 and later[/-] [+]Memory leak on executables embedded with 3.12 and later[/+] on Dec 16, 2023neonene commented on Dec 17, 2023 ContributorAuthor More actions
_testembed test_repeated_simple_init shows it doesn't leak anymore
I can also personally confirm that on Windows, including the 32-bit version of Python. nascheme commented on Dec 22, 2023 I think I've come up with a better fix. See gh-113412. neonene commented on Dec 23, 2023 ContributorAuthor More actions nascheme commented on Dec 23, 2023 Your test_repeated_simple_init case leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:
PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_simple_init
Trace log:
https://gist.github.com/nascheme/fac2dd566f09fe3d75c98134f18c6170
E.g.
==1275635== 502,840 bytes in 9,800 blocks are definitely lost in loss record 421 of 422
==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635== by 0x2FBAC3: unicode_decode_utf8 (unicodeobject.c:4701)
==1275635== by 0x2FC017: PyUnicode_DecodeUTF8Stateful (unicodeobject.c:4834)
==1275635== by 0x2F0E38: PyUnicode_FromString (unicodeobject.c:1893)
==1275635== by 0x3135D9: PyUnicode_InternFromString (unicodeobject.c:14943)
==1275635== by 0x289EBA: PyObject_SetAttrString (object.c:1062)
==1275635== by 0x28504F: _add_methods_to_object (moduleobject.c:174)
==1275635== by 0x2857D1: PyModule_FromDefAndSpec2 (moduleobject.c:384)
==1275635== by 0x3FB5C6: create_builtin (import.c:1396)
==1275635== by 0x4002F6: _imp_create_builtin (import.c:3400)
==1275635== by 0x284483: cfunction_vectorcall_O (methodobject.c:512)
==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635== by 0x21B94C: _PyObject_Call (call.c:348)
==1275635== by 0x21BA23: PyObject_Call (call.c:373)
==1275635== by 0x3856F4: _PyEval_EvalFrameDefault (generated_cases.c.h:1250)
==1275635== by 0x37E2B1: _PyEval_EvalFrame (pycore_ceval.h:115)
==1275635== by 0x3AA774: _PyEval_Vector (ceval.c:1771)
And
==1275635== 339,080 bytes in 6,920 blocks are definitely lost in loss record 420 of 422
==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635== by 0x2F11FE: _PyUnicode_FromUCS1 (unicodeobject.c:2025)
==1275635== by 0x2F16F9: PyUnicode_FromKindAndData (unicodeobject.c:2096)
==1275635== by 0x414B4E: r_object (marshal.c:1157)
==1275635== by 0x414DC0: r_object (marshal.c:1222)
==1275635== by 0x415AC6: r_object (marshal.c:1399)
==1275635== by 0x414DC0: r_object (marshal.c:1222)
==1275635== by 0x415A8A: r_object (marshal.c:1393)
==1275635== by 0x416028: read_object (marshal.c:1524)
==1275635== by 0x416339: PyMarshal_ReadObjectFromString (marshal.c:1641)
==1275635== by 0x3FCDF0: unmarshal_frozen_code (import.c:2062)
==1275635== by 0x4008B1: _imp_get_frozen_object_impl (import.c:3575)
==1275635== by 0x3F8E6B: _imp_get_frozen_object (import.c.h:282)
==1275635== by 0x284098: cfunction_vectorcall_FASTCALL (methodobject.c:425)
==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635== by 0x21B94C: _PyObject_Call (call.c:348)
==1275635== by 0x21BA23: PyObject_Call (call.c:373)
nascheme commented on Dec 23, 2023 The following change seems to greatly reduce the leaks. The _Py_hashtable_set(INTERNED_STRINGS, s, s) call looks like a problem to me.
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -14870,6 +14870,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p)
return;
}
+#if 0
/* Look in the global cache first. */
PyObject *r = (PyObject *)_Py_hashtable_get(INTERNED_STRINGS, s);
if (r != NULL && r != s) {
@@ -14885,6 +14886,7 @@ _PyUnicode_InternInPlace(PyInterpreterState *interp, PyObject **p)
}
return;
}
+#endif
/* Look in the per-interpreter cache. */
PyObject *interned = get_interned_dict(interp);
neonene commented on Dec 24, 2023 ContributorAuthor More actions My memory leak seems to come from _PyUnicode_ClearInterned() when the Py_DEBUG flag is not specified. I'm not sure of the root cause, though.
|
_PyUnicode_ClearInterned(PyInterpreterState *interp) |
|
{ |
|
PyObject *interned = get_interned_dict(interp); |
|
if (interned == NULL) { |
|
return; |
|
} |
|
assert(PyDict_CheckExact(interned)); |
|
|
|
/* TODO: |
|
* Currently, the runtime is not able to guarantee that it can exit without |
|
* allocations that carry over to a future initialization of Python within |
|
* the same process. i.e: |
|
* ./python -X showrefcount -c 'import itertools' |
|
* [237 refs, 237 blocks] |
|
* |
|
* Therefore, this should remain disabled for until there is a strict guarantee |
|
* that no memory will be left after `Py_Finalize`. |
|
*/ |
|
#ifdef Py_DEBUG |
|
/* For all non-singleton interned strings, restore the two valid references |
|
to that instance from within the intern string dictionary and let the |
|
normal reference counting process clean up these instances. */ |
neonene commented on Dec 29, 2023 ContributorAuthor More actions For the record, my memory gain reduced from 2M bytes to 200K bytes per cycle with gh-113412 (non-debug).
Regarding the crash after re-initialization due to the full cleanup by _PyUnicode_ClearInterned():
|
main
(leaks) |
#113218 |
#113412 |
| datetime (current) |
crash |
crash |
crash |
| datetime (isolated) |
|
|
|
| ctypes |
crash |
crash |
crash |
| _ctypes |
crash? |
crash |
|
| _lsprof |
|
|
|
| faulthandler |
|
|
|
| tracemalloc |
|
|
|
| _testcapi |
|
crash |
|
A dangling pointer in a non-isolable (small) extension could be worked around by making the interned string statically allocated, if any.
Original issue: #100911 nascheme commented on Dec 29, 2023 My previous comment about disabling the INTERNED_STRINGS global hash table was incorrect. I must have built with Py_DEBUG enabled and so I erroneously thought it fixed the leak. You seem to be correct about _PyUnicode_ClearInterned(). There is this comment:
/* TODO:
* Currently, the runtime is not able to guarantee that it can exit without
* allocations that carry over to a future initialization of Python within
* the same process. i.e:
* ./python -X showrefcount -c 'import itertools'
* [237 refs, 237 blocks]
*
* Therefore, this should remain disabled for until there is a strict guarantee
* that no memory will be left after `Py_Finalize`.
If Py_DEBUG is enabled, then that function will free interned strings with the flag SSTATE_INTERNED_IMMORTAL. That comment was from Eddie's immortal refcnt change (gh-19474, ea2c001). changed the title [-]Memory leak on executables embedded with 3.12 and later[/-] [+]Memory leak on executables embedded with 3.13[/+] on Oct 15, 2024neonene commented on Oct 15, 2024 ContributorAuthor More actions I think this can be closed after 3203a74 is backported to 3.13. #113190 is opened for 3.12. erlend-aasland commented on Oct 15, 2024 nascheme commented on Oct 15, 2024 GH-118618 would fix the most serious memory leak in 3.12. For the last bugfix release of 3.12, @Yhg1s decided it was too large/disruptive to merge a backport and so that PR is closed. Assuming we are not changing our minds on merging that, I think the memory leak due to the obmalloc state will remain unfixed in 3.12. In theory there might be a less invasive and simpler fix for 3.12, I'd have to research that though.
It's getting confusing to keep track of these embedding bugs and what fixes have been backported. There are two classes of them: memory leaks and crashers. The crashers are the more serious ones. I believe we have fixes the known crashers (see GH-124865, GH-124536, GH-125314). For the memory leaks, the obmalloc state one is the worst. The leaks due to interned strings also depend on the mortal string change (49f6beb). Running the 3.12 head with ASAN enabled would show what's still leaking.
In the long term, i think it would be nice to have CPython running with no ASAN warnings. Then we have a build-bot to ensure we don't regress on that. Reacted by Erlend E. Aasland and Hu Yueh-Wei Yhg1s commented on Oct 21, 2024 FYI, 3.12.7 was not the last bugfix release of 3.12. We have scheduled bugfix releases of 3.12 until April 2025. vstinner commented on Nov 27, 2024 I cannot reproduce this issue anymore on the main branch:
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d15dd519dbf..3f9455a197f 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -38,7 +38,7 @@ char **main_argv;
#define PROGRAM_NAME L"./_testembed"
#define PROGRAM_NAME_UTF8 "./_testembed"
-#define INIT_LOOPS 4
+#define INIT_LOOPS 100
// Ignore Py_DEPRECATED() compiler warnings: deprecated functions are
// tested on purpose here.
@@ -220,6 +220,10 @@ static int test_repeated_simple_init(void)
_testembed_Py_Initialize();
Py_Finalize();
printf("Finalized\n"); // Give test_embed some output to check
+
+ char cmd[1000];
+ sprintf(cmd, "grep ^VmRSS /proc/%i/status", getpid());
+ system(cmd);
}
return 0;
}Output:
--- Loop #1 ---
Finalized
VmRSS: 9424 kB
--- Loop #2 ---
Finalized
VmRSS: 9932 kB
--- Loop #3 ---
Finalized
VmRSS: 9952 kB
(...)
--- Loop #98 ---
Finalized
VmRSS: 9588 kB
--- Loop #99 ---
Finalized
VmRSS: 9588 kB
--- Loop #100 ---
Finalized
VmRSS: 9576 kB
Since 3.12 branch is not going to be fixed, can we close the issue?
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|