GeezerVison is showing github.com — ENH: Please support subinterpreters · Issue #24755 · numpy/numpy
ENH: Please support subinterpreters #24755DescriptionProposed new feature or change:
Version 1.25.1, Python 3.12.re02
After enabling interpreters in Python C Api:
PyInterpreterConfig config = {
.check_multi_interp_extensions = 1,
.gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
if (PyStatus_Exception(status)) {
return -1;
}
Import numpy throws an exception:
module numpy.core._multiarray._umath does not support loading in subinterpreters Reacted by VilleKrumlinde, dawmster, AlexisM, Danny Kim, Lance Legel, Ralph J. Steinhagen, Donghyun-Edit, Alexander Neundorf, Richard Lindsley, Almar Klein and 3 more Activitychanged the title [-]Support for subinterpreters[/-] [+]ENH: Please support subinterpreters[/+] on Sep 21, 2023mattip commented on Sep 21, 2023 PEP 554 states:
To mitigate that impact and accelerate compatibility, we will do the following:
- be clear that extension modules are not required to support use in multiple interpreters
- raise ImportError when an incompatible module is imported in a subinterpreter
- provide resources (e.g. docs) to help maintainers reach compatibility
- reach out to the maintainers of Cython and of the most used extension modules (on PyPI) to get feedback and possibly provide assistance
The PEP also links to Isolating Extensions which has a lot of theory, but does not clearly state how to migrate a large existing c-extension library like NumPy to support subinterpreters. I think we would need to:
- move to HeapTypes
- move all static state into module state
- carefully analyze code for possible shared state.
I am a bit unclear whether subinterpreters share a single GIL, if not we would also have to carefully examine the code for possible race conditions.
This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort? rgommers commented on Sep 21, 2023 Relevant mailing list threads and issues:
seberg commented on Sep 26, 2023
This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort?
I suspect the vast majority of changes to be relatively easy, but there is still the same problem that we need someone to explicitly dedicate time on this, and I doubt it will be one of the current core devs.
We even added a warning a long time back saying exactly that, but it seems CPython changes to make subinterpreter support better in the long-run now enforces an error rather than a warning. a-reich commented on Oct 10, 2023
PEP 554 states: …
FWIW the recent CPython changes should be from PEP 684 “Per-Interpreter GIL”; PEP 554, for the Python API and subinterpreter management features, is still in draft status. mdekstrand commented on Nov 10, 2023 There's a very strong use case for subinterpreters since PEP 684 for parallel processing that I expect would be useful to a lot of numpy client code: using subinterpreters in separate threads will enable shared memory (at least in a read-only case) with significantly less hassle than multiprocessing. Reacted by AlexisM, Danny Kim, Stella Laurenzo and Mike R. Reacted by a-reich a-reich commented on Nov 10, 2023 I’m also very excited about the potential opportunities of using subinterpreters with numpy, and agree with what @mdekstrand said. In particular, the latest draft of PEP 734 discusses sharing data via the buffer protocol (and already implemented in the private interpreters module since 3.13a1). Since ndarrays can export their buffer or be created from one without copies, this could be a very nice pattern:
- pickle your array with protocol 5 to get some serialized metadata plus the memoryview,
- pass that view to a bunch of interpreters (which is basically instant) as well as the small metadata,
- and unpickle: now all of them are sharing the data in each of their arrays
- And if you don’t want to worry about data races, seems like np can handle that by setting the readonly flag.
You get concurrency with performant, opt-in data sharing, without the hassles of managing subprocesses and using multiprocessing.shared_memory where you have to create a shared buffer of fixed size ahead of time and only create arrays using that. With interpreters you can take any random array you got and easily share it. paultiq commented on Oct 23, 2024 InterpreterPoolExecutor's are to be introduced in 3.14 124548.
At present, numpy imports fail due to "ImportError: module numpy._core._multiarray_umath does not support loading in subinterpreters":
See following example: TPE and PPE work, IPE does not.
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor
from interpreters_backport.concurrent.futures.interpreter import InterpreterPoolExecutor
def try_tpe():
with ThreadPoolExecutor() as executor:
executor.submit(exec, "import numpy as np;print('TPE:', np.random.rand(2))")
def try_ppe():
with ProcessPoolExecutor() as executor:
executor.submit(exec, "import numpy as np;print('PPE:', np.random.rand(2))")
def try_ipe():
with InterpreterPoolExecutor() as executor:
f = executor.submit(exec, "import numpy as np;print('IPE:', np.random.rand(2))")
f.result()
if __name__ == "__main__":
try_tpe()
try_ppe()
try_ipe()Footnote: I noted this recent comment, which perhaps closed the door on hope here; #27192 (comment) temeddix commented on Oct 30, 2024 It's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with InterpreterPoolExecutor and Numpy will not be able to deal with new demands.. rgommers commented on Oct 30, 2024
It's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with InterpreterPoolExecutor and Numpy will not be able to deal with new demands..
They aren't closed. A lot of work has happened (and is still happening) to supported free-threaded CPython over the past 6 months. Pretty much all that work is also directly relevant to support for subinterpreters. The extra work needed for subinterpreters - primarily moving to heap types I believe - will need someone to dig in and do that work though. None of the active maintainers are working on this, however contributions are very much welcome. Reacted by AlexisM Reacted by Danny Kim, Amit Singh and Julianus Pfeuffer mattip commented on Oct 30, 2024 The list of tasks is still this, although the documentation has gotten better:
- move all static types to HeapTypes (i.e. search for
static PyTypeObject and replace that with PyType_FromModuleAndSpec() or some other heap allocation of the type. This can be done one type at a time, and must be benchmarked for performance implications.
- move all static state into module state. Much of the groundwork for this has been done in the free-threading code fixes. Again, better to go in small increments and to benchmark for performance implications.
- carefully analyze code for other possible shared state.
paultiq commented on Oct 30, 2024 mattip commented on Oct 30, 2024 Yes, that is part of the "carefully analyze code for other possible shared state". Cython does support the constructs needed for generating code compatible with subinterpreters, the question is more "are we using unsafe coding practices in our cython code?"
Another area that will need adaptation is the f2py generated wrappers. AraHaan commented on May 15, 2025
The list of tasks is still this, although the documentation has gotten better:
* move all [static types to HeapTypes](https://docs.python.org/3/howto/isolating-extensions.html#changing-static-types-to-heap-types) (i.e. search for `static PyTypeObject` and replace that with [`PyType_FromModuleAndSpec()`](https://docs.python.org/3/c-api/type.html#c.PyType_FromModuleAndSpec) or some other heap allocation of the type. This can be done one type at a time, and must be benchmarked for performance implications.
* move all [static state into module state](https://docs.python.org/3/howto/isolating-extensions.html#module-state-access-from-functions). Much of the groundwork for this has been done in the free-threading code fixes. Again, better to go in small increments and to benchmark for performance implications.
* carefully analyze code for other possible shared state.
With that it is then theoretically possible to compile numpy with Py_LIMITED_API=[hex value for 3.14 here] But then one cannot use the free-threaded define yet with the limited API (😭).
In fact, I think the day when both defines are allowed it would be the best day for Numpy as then single builds from that version on could be built with that version minimally with both of those defines and work out-of-box in all the future versions of CPython provided they do not extend the size of the PyObject structure and cause a warning to display when loading the C extensions. seberg commented on May 15, 2025
With that it is then theoretically possible to compile numpy with Py_LIMITED_API=[hex value for 3.14 here]
I don't think that is correct or necessary. Neither is a possible change of the struct any issue at all for NumPy as we don't use the limited API. marcan commented on Aug 22, 2025 Just wanted to drop a reference to python/cpython#138045 here. This is a more subtle issue: Due to a Python 3.13 regression, numpy currently crashes on load inside a subinterpreter, even if that subinterpreter is the only one loading numpy and it is not using full isolation (that's the .check_multi_interp_extensions = 1 from OP, which would be 0 in this case).
I believe this previously worked, as long as only a single subinterpreter ever imports numpy.
I also believe the crash/badness happens even if you configure things like OP, requesting full isolation. That would previously fail cleanly with an exception, but now causes interpreter state corruption.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|