GeezerVison

Current documentation:

Set an initialization callback. It allows executing code as soon as the
Python interpreter is initialized, before the first import. For example, it
can be used to add a meta path importer into sys.meta_path.

Python is not fully initialized yet when the callback is called. For
example, sys.stdout may not exist yet.

@indygreg: Hi, would such C API fit your needs for PyOxidizer?

cc @encukou @serhiy-storchaka

Python is not fully initialized yet when the callback is called. For example, sys.stdout may not exist yet.

That is extremely vague. If there are no concrete promises about what's available, and what APIs can be called, new versions of CPython are theoretically allowed to "break" callers.
For example: what can you import at this point?

Is the callback better than restoring older private "multi-phase" API?
Should it be unstable?
Is it possible that we'll need to add more callbacks at different points of the initialization?

Would it be better to add API specifically for adding a simple metapath entry?

That is extremely vague. If there are no concrete promises about what's available, and what APIs can be called, new versions of CPython are theoretically allowed to "break" callers.

It's vague on purpose, but you're right that it should be better specified.

Python 3.13 documentation specify what is available at the "core" phase and what is the "main" phase: https://docs.python.org/3.13/c-api/init_config.html#multi-phase-initialization-private-provisional-api

For example: what can you import at this point?

Good question. I just tested: you can import built-in modules at this point, but only built-in modules. importlib is not fully initialized yet.

Is the callback better than restoring older private "multi-phase" API?

The old API (Python 3.13 and older) allows not executing the "main" initialization phase, but IMO it makes no sense (there is not use case for that). What's needed is only to inject code (callback) between the "core" and the "main" initialization phase. IMO a callback API better fits the use case.

Is it possible that we'll need to add more callbacks at different points of the initialization?

I'm not aware of other points which would be interesting during the initialization.

Would it be better to add API specifically for adding a simple metapath entry?

The bypy code to inject the sys.meta_path hook is non-trivial:

https://github.com/kovidgoyal/bypy/blob/c53e47497c5cb3ae9d7eb640b93a28a64fddb96a/bypy/freeze/bypy-freeze.h#L753-L783

The PyOxidizer code is also complex:

https://github.com/indygreg/PyOxidizer/blob/1ceca8664c71f39e849ce4873e00d821504b32bd/pyembed/src/interpreter.rs#L264-L316

Python 3.13 documentation specify what is available at the "core" phase and what is the "main" phase: https://docs.python.org/3.13/c-api/init_config.html#multi-phase-initialization-private-provisional-api

I completed the documentation using Python 3.13 doc.

Good question. I just tested: you can import built-in modules at this point, but only built-in modules. importlib is not fully initialized yet.

That's worrying: it means that if we rewrite a module in Python, we break people's code. (For example, the bypy hook depends on marshal being built-in. In this case I guess they could switch to PyMarshal_ReadObjectFromString, but, that's still a code change...)

I completed the documentation using Python 3.13 doc.

What's worrying for metapath entries is that "importlib", "Path Configuration" and "sys.path initialization" come after the hook call. It sounds like any change in this are could break users of the hook.

This does look like it should be unstable at least.

But, I believe it would be best to bring the old private API back (deprecated) while a better alternative is designed:

• I don't see the callback being a substantially better design, TBH.
• Since the API was specified in a PEP, and was added before PEP-387 & PEP-689 which add better processes than "provisional" underscore-prefixed APIs, I think it should have had a deprecation period.

cc @ericsnowcurrently

What's worrying for metapath entries is that "importlib", "Path Configuration" and "sys.path initialization" come after the hook call. It sounds like any change in this are could break users of the hook.

These things changed multiple times since PEP 587 (PyConfig C API) has been implemented, and it was just fine with users of the "Multi-Phase Initialization Private Provisional API". For example, the path configuration has been reimplemented in Python in Python 3.11 (Modules/getpath.py).

Since the API was specified in a PEP, and was added before PEP-387 & PEP-689 which add better processes than "provisional" underscore-prefixed APIs, I think it should have had a deprecation period.

Well, a provisional API has a different backward compatibility contract: https://docs.python.org/dev/glossary.html#term-provisional-API

A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees.

Anyway, the API has already been removed from Python 3.14.

A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees.

Anyway, the API has already been removed from Python 3.14.

It also says:

Such changes will not be made gratuitously – they will occur only if serious fundamental flaws are uncovered that were missed prior to the inclusion of the API.

What are the serious fundamental flaws that were missed?

That's worrying: it means that if we rewrite a module in Python, we break people's code. (...)

and

What's worrying for metapath entries is that "importlib", "Path Configuration" and "sys.path initialization" come after the hook call. It sounds like any change in this are could break users of the hook.

These issues already existed in Python 3.13 with the provisional _init_main=0 / _Py_InitializeMain() API. Proposed PyInitConfig_SetInitCallback() has the same issues than adding back _Py_InitializeMain() to Python 3.15. As I wrote previously, it's not an issue in practice, and this code already changed multiple times without breaking projects using _Py_InitializeMain().

But, I believe it would be best to bring the old private API back (deprecated) while a better alternative is designed

I removed the _Py_InitializeMain() API because I failed to find any user of this provisional/private API. But there are at least two users in the wild (bypy and PyOxidizer), so I would prefer to add a documented and tested public API to replace it.

I don't see the callback being a substantially better design, TBH.

Python 3.13 API allows to not execute the "main" initialization phase. This is not needed by 3rd party projects and I would like to avoid that. PyInitConfig_SetInitCallback() ensures that Python is always initialized the same way (always execute the "main" initialization phase).

These issues already existed in Python 3.13 with the provisional _init_main=0 / _Py_InitializeMain() API. Proposed PyInitConfig_SetInitCallback() has the same issues than adding back _Py_InitializeMain() to Python 3.15.

Right.
My question is: how is the new proposed API better? Is it worth breaking projects?

Python 3.13 API allows to not execute the "main" initialization phase.

Is that the only benefit?

I removed the _Py_InitializeMain() API because I failed to find any user of this provisional/private API. But there are at least two users in the wild (bypy and PyOxidizer), so I would prefer to add a documented and tested public API to replace it.

Where did you look?
PyPI is not a very good place to find users of embedding API & runtime initialization.

I propose replacing _Py_InitializeMain() with PyInitConfig_SetInitCallback() because I don't want to expose the concepts of "core initialization" and "main initialization" in the API. A callback is easier to document and explain.

Where did you look?

I don't recall, but I likely looked in PyPI top 15,000 projects.

I don't want to expose the concepts of "core initialization" and "main initialization" in the API.

Fair. I don't think that qualifies as a serious fundamental flaw though.

I likely looked in PyPI top 15,000 projects.

Yeah. The extension module index isn't a good place for users of runtime initialization API.

I don't want to expose the concepts of "core initialization" and "main initialization" in the API

I think we should, though not necessarily under these names.

There's no reasons why embedders shouldn't be able to "initialize up to pre-importlib", do stuff, and then "initialize importlib and anything after it". That's more or less what we currently do, where the "do stuff" is running getpath.py.

It would also help our own tools, where we need to half-initialize the runtime in order to freeze modules during build, etc.

It might not directly fix the problem being looked at here, but it's the design we've been aiming towards for a decade now, so it ought to be the goal. If that means bringing back the API that was removed in the meantime, that seems fine. The only thing we gain by rushing initialisation API changes is cruft, so we're better off keeping the cruft we already have and getting the API design right.

Oh, I'll also point out that I think it makes perfect structural sense for getpath.py to belong to python.exe rather than libpython.so, which would imply that it's possible to do the three steps using only the public API.

For practical reasons, we wouldn't make that a real move, but structurally I think it helps to visualise importlib as part of "Python the command-line tool" rather than "Python the embeddable library" (since when embedding, you often want a different kind of importer than importlib's defaults).

@encukou and @zooba dislike the callback approach (PR gh-142420), so I wrote PR gh-143877 to add Py_InitializeMain() function and rename PyConfig._init_main to PyConfig.init_main.

Oh, I'll also point out that I think it makes perfect structural sense for getpath.py to belong to python.exe rather than libpython.so, which would imply that it's possible to do the three steps using only the public API.

For practical reasons, we wouldn't make that a real move, but structurally I think it helps to visualise importlib as part of "Python the command-line tool" rather than "Python the embeddable library" (since when embedding, you often want a different kind of importer than importlib's defaults).

Unfortunately, I think that would break a number of use-cases, but we could definitely move the C API towards that.

Unfortunately, I think that would break a number of use-cases, but we could definitely move the C API towards that

I assume this is agreeing with the half of my post saying that we couldn't actually remove getpath from libpython? It's unclear whether you're agreeing that removing getpath would break people who assume it'll be there or not, since more than half the quoted text is saying that it would break people if we actually did it.

I updated my PR to address @gpshead's review.

Closed in favor of PR gh-145472.