GeezerVison is showing github.com — Please restore _Py_InitializeMain or provide an alternate API for it · Issue #142417 · python/cpython
Please restore _Py_InitializeMain or provide an alternate API for it #142417DescriptionAs discussed in #129033, opening a new issue at the request of @vstinner. See #129033 (comment) for the reasons this API is needed.
In short, a way to run python code and insert a meta importer into sys.meta_path before the "main" phase of initialization runs is needed. This allows, for instance, to bundle all python code as pre-compiled bytecode into a single mmapped file and directly load it from there for improved performance and hygiene.
I am aware that the state of the interpreter after "core" initialization is not entirely specified, however, in practice it is complete enough to write a meta path importer.
This facility has been used for many years by at least two projects that I know of: bypy (link to source code) https://github.com/kovidgoyal/bypy/blob/master/bypy/freeze/bypy-freeze.h#L835 and PyOxidizer: https://github.com/indygreg/PyOxidizer/blob/1ceca8664c71f39e849ce4873e00d821504b32bd/pyembed/src/technotes.rs#L155
I dont much care about this API in particular, but I would like some way to hook into the initialization process after "core" initialization and before "main" initialization.
Some context about use case. bypy is used in the calibre and kitty programs, both of which embed a "frozen" Python interpreter. _Py_InitializeMain() allows that frozen importer to import all its python code, including stdlib code from a single mmaped file contianing pre-compiled bytecode. This has obvious performance benefits especially on windows.
Linked PRs
Activityvstinner commented on Dec 8, 2025 If this feature is used, we should add a clean C API to use it, rather than just restore the private API to move back to the status quo.
I dislike config._init_main=0 to stop the initialization at the "core" phase, and then having to call _Py_InitializeMain() to "finish" the initialization to the main phase.
IMO a simple callback would be enough for bypy and PyOxidizer use cases: it would be called between core and main initialization phases. In practice, it would be called once the Python interpreter is initialized and before the first import. Something like:
PyStatus PyConfig_SetInitCallback(PyStatus (*callback)(void *arg), void *arg);
int PyInitConfig_SetInitCallback(PyInitConfig *config, PyStatus (*callback)(void *arg), void *arg); @kovidgoyal: Would it fit your use cases?
cc @ncoghlan @ericsnowcurrently vstinner commented on Dec 8, 2025 For the 3.14 branch, we can restore the private _Py_InitializeMain() function, but only in Python 3.14.3. It means that Python 3.14.0, 3.14.1 and 3.14.2 cannot be used. kovidgoyal commented on Dec 8, 2025 Yes, I can make that work in bypy. It will require a little bit of
re-writing, but no big deal. Far more important for me is to have an
official stable API for this. Whatever shape that API takes I can adapt
to it, provided it allows me to run my callback immediately before the
first import but after the interpreter is initialized. Your proposed
PyConfig_SetInitCallback() would be fine for me.
Personally, I am anyway patching Python 3.14 currently to restore that
function (since I embed private python in my apps, that's easy to do).
So to me it does not matter what happens in 3.14 lifecycle or later. I
cant speak for other projects, of course. Reacted by Victor Stinner vstinner commented on Jan 15, 2026 @encukou and @zooba dislike the callback approach (PR gh-142420), so I wrote PR gh-143877 to add Py_InitializeMain() function and rename PyConfig._init_main to PyConfig.init_main. kovidgoyal commented on Jan 15, 2026 zooba commented on Jan 15, 2026 For the record, I also dislike adding it as a public API, and would prefer to just restore the private one until we've designed initialization properly. Reacted by Petr Viktorin and Alyssa Coghlan vstinner commented on Feb 6, 2026 @zooba:
For the record, I also dislike adding it as a public API, and would prefer to just restore the private one until we've designed initialization properly.
What do you mean by designing initialization properly? IMO the initialization callback is step towards a better initialization design (PR gh-142420).
IMO adding back the private API is step backwards, I would prefer avoid that. zooba commented on Feb 6, 2026 "A step" is not the better design. Do the entire design and let's discuss it as a whole, and if we all agree that it's better overall, then we'll implement the whole design.
Until then, we should preserve what people are currently using. Reacted by Alyssa Coghlan ncoghlan commented on Feb 23, 2026 I agree with @zooba on this one (unbreak users first, then add the cleaner design after discussing it without time pressure, then potentially remove the legacy private API only after folks consuming that private API have had the chance to add the relevant conditional usage of the new API).
While I think we can probably do something cleaner with a public unstable API that exposes the relevant substeps in a nicer way than the current private API, exposing this stage of the startup as a regular public API seems like a bad idea (I know I proposed that myself way back in PEP 432, but there are a bunch of good reasons we never actually implemented that design, and these days I consider this one of them). vstinner commented on Mar 3, 2026 The majority prefers restoring the private _Py_InitializeMain() function, rather than my proposed public C APIs: @encukou (comment), @zooba (comment) and @ncoghlan (comment).
So I wrote PR gh-145472 to restore the private _Py_InitializeMain() function in Python 3.15. vstinner commented on Mar 4, 2026
Please restore _Py_InitializeMain or provide an alternate API for it
The change 31343cf restores _Py_InitializeMain() function.
I close the issue. kovidgoyal commented on Mar 4, 2026 Reacted by Victor Stinner encukou commented on Mar 4, 2026
The majority prefers restoring the private _Py_InitializeMain() function, rather than my proposed public C APIs
Well, I see your proposal as a first draft, which now has time to growr. Hopefully we can make things better and not break users.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|