GeezerVison

Per the benchmarks, this change is a little faster (less than 1%) on Linux/GCC.

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit feb8ef5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

It might be worth trying to see what is the performance impact of storing the interpreter state in TLS as well.
No need to do that in this PR, though.

FTR, on Windows this introduced a ~2% performance regression, and on MacOS there's ~3% regression.

Note that these penalties may be partially mitigated by passing the current thread state as an argument throughout the internal C-API (where currently we only do so in some places). The implementation here is also relatively naïve. There are likely opportunities to improve performance via compiler-specific directives.

(While this change was mostly done independently, I did take some inspiration from earlier (~2020) work by @markshannon (main...markshannon:threadstate_in_tls) and @vstinner (#23976).)

I prepared this change for years in advance:

• Python 3.8:

  • I modified the PyThreadState_GET() macro to make it an alias to PyThreadState_Get() function: so it's always a function call, and no longer a macro to hide implementation details.
  • I added _PyThreadState_GET() static inline function to the internal C API to get more freedom on its implementation.
  • I modified many internal C functions to pass explicitly the tstate variable: Pass the Python thread state explicitly. For example, I added _PyErr_Occurred(tstate) function which replaces PyErr_Occurred() (which has no argument). The idea is that in the future, calling _PyThreadState_GET() may before slower if the value is retreived from a thread local storage (TLS) variable: which is done in this PR.
  • I modified many functions to pass the "state" more explicitly: runtime, tstate and/or interp. Example: _PySys_Create(runtime, interp, &sysmod) call in Python/pylifecycle.c.

• Python 3.9:

  • I converted _PyThreadState_GET() macro to a static inline function.
  • I added _PyInterpreterState_GET() static inline function. Maybe the implementation will change in the future to be more efficient (ex: thread local storage?).

My only remaining uncertainty is with the existing "GIL is held" constraint. With _PyRuntime.tstate_current, it was only guaranteed valid in the thread currently holding the GIL, if any. With this change, it is valid even when the GIL isn't held. I don't see how that would be a problem, but I'm going to double-check anyway.

In the Python 3.9 and 3.10 era, I moved multiple global states to the "interpreter state" (interp): https://pythondev.readthedocs.io/subinterpreters.html#done These changes caused various crashes in third party C extensions which use the C API with the GIL released (!). For example, calling PyLong_FromLong(1) with the GIL released. This was always illegal and invalid according to the C API documentation. But you know, there are always bugs in the wild. All affected C extensions have been fixed in the meanwhile. Also, some states were made global again (small integer singletons), and immortal objects also made the situation differnet.

FTR, on Windows this introduced a ~2% performance regression, and on MacOS there's ~3% regression.

It might be interesting to check the hot code calling _PyThreadState_GET() and see if tstate could be passed to only call _PyThreadState_GET() once. I'm not sure if it's worth it. Also, in stdlib C extensions, I would prefer to use the internal C API less rather than more :-)

Does this change fix indirectly the PyGILState API for subinterpreters? See: #59956

Thanks @ericsnowcurrently for taking care of this very old project!

It seems like the !defined(Py_BUILD_CORE_MODULE) test in pycore_pystate.h avoids the complicated linker issuses that I had on Windows and macOS when I tried a similar change in 2020 (PR #23976).