Description

VestniK

opened on Aug 28, 2025

Issue body actions

We are integrating Python as an embedded interpreter in a C++ application to provide a plugin interface for end users. Our codebase is transitioning to a multi-threaded architecture, where each independent unit of work utilizes its own subinterpreter with a separate GIL. This functionality became available in Python 3.12.

We have modified the bundled version of python-rapidjson to ensure compatibility with our setup. The modified version can be found here: https://github.com/yandex/yatool/blob/main/devtools/ymake/contrib/python-rapidjson/rapidjson.cpp

I want to refactor and upstream the subinterpreters compatibility changes in several PRs, ensuring clean and maintainable code.

Are you interested in this feature?

Note: I'm aware that some projects consider subinterpreters to be not fully mature for production use. Therefore, I would like to assess the community’s interest before proceeding with the development.

Activity

1. lelit commented on Aug 30, 2025

   

   lelit

   on Aug 30, 2025

   Contributor

   More actions

   This is similar to issue #225: I bet that as new core Python features become mainstream, there will an increasing interest in having them supported by 3rd party packages, so yes, I think it is sensible, at least when the required changes are not (too) disruptive, well tested and the performance impact is taken in consideration.

   OTOH, for the foreseeable future I won't have a direct motivation to use those features myself, so the best thing would be if an actual user could lead the transition, and you seem a good candidate here 😉

   Thanks in advance!
2. VestniK commented on Sep 3, 2025

   

   VestniK

   on Sep 3, 2025

   Author

   More actions

   I've sent the first PR to get basic feedback on coding style or get some other important notices on what fits the module code base and what does not.

   The roadmap looks like:

   • Turn all static types into heap types.
   • Move all cached PyObjects and other mutable global state into a single struct.
   • Move the struct from the previous step into the rapidjson module.
   • Write multi-threaded subinterpreters test application (which can be compiled and run with thread-sanitizer as a CI pipeline).
   • Finally mark the module per subinterpreter GIL compatible and fix all of the issues form running test from previous step under thread sanitizer.
3. lelit commented on Sep 14, 2025

   

   lelit

   on Sep 14, 2025

   Contributor

   More actions

   Sorry for the lag, enjoyed a short vacation 😉
   Thank you, I will inspect your contribution as soon as possible.
4. lelit commented on Aug 7, 2026

   

   lelit

   on Aug 7, 2026

   Contributor

   More actions

   Sorry for the lag, enjoyed a short vacation 😉 Thank you, I will inspect your contribution as soon as possible.

   ... and then completely forgot about this, sigh! Sorry, I will try to re-focus.
5. lelit commented on Nov 1, 2026

   

   lelit

   on Nov 1, 2026

   Contributor

   More actions

   I think that the sub-interpreters branch (that I rebased on top of current master) brings PEP-684 support.
   As said, it is rather new territory for me (well, IIRC I used proto-sub-interpreters in PyApache, eons ago... 😉), so I'd really appreciate any feedback...

Issue actions