GeezerVison is showing github.com — Legacy-mode subinterpreters in Python 3.12: `import _tkinter` leads to shutdown crash · Issue #115649 · python/cpython
Legacy-mode subinterpreters in Python 3.12: import _tkinter leads to shutdown crash #115649DescriptionBug report
Bug description:
Hello everyone,
I am working on an application embedding multiple Python subinterpreters - for which the Python version should be upgraded from Python 3.10 to Python 3.12.1. For the time being, the "legacy version" of subinterpreters (i.e., using a global, shared GIL) should be used, since all Python extensions (including _tkinter and all extensions with single-phase initialization) should be supported.
If I understand the docs correctly, using the legacy Py_NewInterpreter() method should preserve the existing behavior. Still, the following application crashes at shutdown:
#include <Python.h>
class PythonInterpreter {
public:
PythonInterpreter(PyThreadState* parent, int id)
: mParent(parent)
, mId(id) {
PyEval_RestoreThread(mParent);
mThread = Py_NewInterpreter();
PyObject* globals = PyModule_GetDict(PyImport_AddModule("__main__"));
PyRun_String("import _tkinter", Py_single_input, globals, globals);
PyEval_SaveThread();
std::cout << "Subinterpreter " << id << " done" << std::endl;
}
virtual ~PythonInterpreter() {
std::cout << "destructor " << mId << std::endl;
PyThreadState_Swap(mThread);
Py_EndInterpreter(mThread);
}
private:
PyThreadState* mParent;
PyThreadState* mThread;
int mId;
};
int main(int /* argc */, char** /* argv[] */) {
PyConfig config;
PyConfig_InitPythonConfig(&config);
PyStatus status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
PyConfig_Clear(&config);
Py_ExitStatusException(status);
} else {
PyConfig_Clear(&config);
}
PyThreadState* s0 = PyThreadState_Get();
PyEval_SaveThread();
PythonInterpreter* i0 = new PythonInterpreter(s0, 0);
PythonInterpreter* i1 = new PythonInterpreter(s0, 1);
delete i0;
delete i1;
PyEval_RestoreThread(s0);
Py_Finalize();
}When compiling and running this program with a debug build of Python 3.12.1 (or later) on Linux, I get this output:
Subinterpreter 0 done
Subinterpreter 1 done
destructor 0
destructor 1
main: Objects/dictobject.c:283: unicode_get_hash: Assertion `Py_IS_TYPE(((PyObject*)(((o)))), (&PyUnicode_Type))' failed.
With a non-debug build, the program exits with a segmentation fault.
The gdb backtrace looks as follows:
#0 0x00007ffff6311387 in raise () from /lib64/libc.so.6
#1 0x00007ffff6312a78 in abort () from /lib64/libc.so.6
#2 0x00007ffff630a1a6 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007ffff630a252 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffff6a9baf5 in unicode_get_hash (o=<optimized out>) at Objects/dictobject.c:2143
#5 _PyDict_Next (op=op@entry=0x7fffecfd2270, ppos=ppos@entry=0x7fffffffd2a8, pkey=pkey@entry=0x7fffffffd2a0, pvalue=pvalue@entry=0x7fffffffd298,
phash=phash@entry=0x0) at Objects/dictobject.c:2142
#6 0x00007ffff6a9c0d8 in PyDict_Next (op=op@entry=0x7fffecfd2270, ppos=ppos@entry=0x7fffffffd2a8, pkey=pkey@entry=0x7fffffffd2a0,
pvalue=pvalue@entry=0x7fffffffd298) at Objects/dictobject.c:2189
#7 0x00007ffff6ab3751 in _PyModule_ClearDict (d=0x7fffecfd2270) at Objects/moduleobject.c:624
#8 0x00007ffff6ab40dd in _PyModule_Clear (m=m@entry=0x7fffed02ca70) at Objects/moduleobject.c:604
#9 0x00007ffff6c11bd4 in finalize_modules_clear_weaklist (interp=interp@entry=0x7fffed03c020, weaklist=weaklist@entry=0x7fffef912da0, verbose=verbose@entry=0)
at Python/pylifecycle.c:1526
#10 0x00007ffff6c125ef in finalize_modules (tstate=tstate@entry=0x7fffed099950) at Python/pylifecycle.c:1609
#11 0x00007ffff6c202da in Py_EndInterpreter (tstate=0x7fffed099950) at Python/pylifecycle.c:2220
#12 0x00000000004014b5 in PythonInterpreter::~PythonInterpreter (this=0x5092f0, __in_chrg=<optimized out>) at main.cc:25
#13 PythonInterpreter::~PythonInterpreter (this=0x5092f0, __in_chrg=<optimized out>) at main.cc:26
#14 0x00000000004012e2 in main () at main.cc:60
Digging further into the backtrace, it looks like the Python garbage collector is trying to decrease the reference counter to the _tkinter module twice, despite it having been increased only once. Oddly enough, the program runs just fine when destroying the interpreters in reverse order:
PythonInterpreter* i0 = new PythonInterpreter(s0, 0);
PythonInterpreter* i1 = new PythonInterpreter(s0, 1);
delete i1;
delete i0;Can anyone help me shed some light into this issue? Is there anything I am overlooking?
CPython versions tested on:
3.12.1, 3.12.2, 3.13.0a4
Operating systems tested on:
Linux, Windows
Linked PRs
Activitycar-bianco commented on Feb 20, 2024 I can reproduce the issue with this Python code.
import _xxsubinterpreters as interpreters
interp0 = interpreters.create(isolated = 0)
interp1 = interpreters.create(isolated = 0)
interpreters.run_string(interp0, 'import _tkinter')
interpreters.run_string(interp1, 'import _tkinter')
interpreters.destroy(interp0)
interpreters.destroy(interp1)
print('Successful!')Funnily enough, swapping the two destroy calls:
interpreters.destroy(interp1)
interpreters.destroy(interp0) makes everything work as intended. tonybaloney commented on Feb 26, 2024 I suspect, since this crash looks very similar to other crashes I've seen doing the same thing like #112292 and #112140 that _tkinter has single-phase init and/or global state so the first time the module is unloaded certain global state objects are deallocated and then the second time it crashes because tuple items refer to things that don't exist.
There's a longer list of them here #112677
Why didn't this crash in older versions? because the sub interpreters implementation wasn't completed until 3.12 and so the state was global and not per-interpreter anyway.
I'm assuming that if you set isolated=1 it will tell you that tkinter isn't supported?
I think you should get that warning as well even with non-isolated sub interpreters since it will cause this type of crash CC @ericsnowcurrently aisk commented on Jun 10, 2024 In the main branch (c3b6dbf), no mater the destroy order, the interpreter will crash, this is a minimal code to reproduce it:
import _interpreters
iid = _interpreters.create()
_interpreters.exec(iid, "import _tkinter")
_interpreters.destroy(iid) $ ./python c.py
Debug memory block at address p=0x7899b89e3520: API '�'
15987178197214944733 bytes originally requested
The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
at p-7: 0xdd *** OUCH
at p-6: 0xdd *** OUCH
at p-5: 0xdd *** OUCH
at p-4: 0xdd *** OUCH
at p-3: 0xdd *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 8 pad bytes at tail=0xddde5677967c12fd are fish: Job 1, './python c.py' terminated by signal SIGSEGV (Address boundary error)
I guess a lot codes have changed, so the original issue may be fixed, but another bug was introduces in #118157. added a commit that references this issue on Jun 17, 2024gh-115649: Copy the filename into main interpreter before intern in i… ericsnowcurrently commented on Jun 18, 2024 Is this still a problem? I just tried to reproduce the crash against main and it worked fine. jan-hasse commented on Jun 19, 2024 Reacted by Eric Snow and An Long
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|