Description

erlend-aasland

opened on May 27, 2022

Issue body actions

As suggested in python/cpython#93012 (comment), we should consider adding PyType_FromMetaclass.

@wjakob already has an implementation, see python/cpython#93012 (comment).

Reacted by Aaron Gokaslan

Activity

1. vstinner commented on May 28, 2022

   

   vstinner

   on May 28, 2022

   Member

   More actions

   What's the use case? Convert a static type to a heap type and be compatible with Python 3.11 and older, or use PyType_FromMetaclass() on Python 3.12 and newer?

   Reacted by Aaron Gokaslan
2. vstinner commented on May 28, 2022

   

   vstinner

   on May 28, 2022

   Member

   More actions

   cc @encukou
3. encukou commented on May 30, 2022

   

   encukou

   on May 30, 2022

   Member

   More actions

   The first. PyType_FromMetaclass() is already merged in main :)

   The function is apparently very useful for language/framework bindings, several of which (PySide, JPype) now use ugly hacks to reimplement its functionality. Or they would need to use hacks if they want to switch to limited API (nanobind).

   Before including it here, please wait for a few more fixes & tests to get sorted out & merged: python/cpython#28748
4. vstinner commented on Jul 21, 2023

   

   vstinner

   on Jul 21, 2023

   Member

   More actions

   I'm not comfortable with adding such function since the implementation is quite big :-( Can projects which need it copy it from CPython?
5. vstinner commented on Jul 27, 2023

   

   vstinner

   on Jul 27, 2023

   Member

   More actions

   When I tried to convert static types to heap types and port C extension to multiphase init API (PEP 489), I saw that storing the module in a type is also a key feature, and it's not supported by Python 3.8 and older, whereas pythoncapi-compat supports Python 3.5 to 3.13.

   Python 3.9 added PyType_FromModuleAndSpec(), PyType_GetModule(), PyType_GetModuleState(), and PyHeapTypeObject.ht_module member.

   PyType_GetModuleByDef() was only added to Python 3.11, whereas it's required to port many C extensions to multiphase init API. PyType_GetModuleByDef() implementation is around 34 lines.

   Backporting these functions to Python 3.8 would require a hash table associating a type to a module (using a weak reference), since there is no API in old Python versions to store arbitrary data in PyTypeObject when creating a new type :-(

   I don't think that pythoncapi_compat.h design is a good fit to implement these features. I would be more comfortable with a library, rather than a header file.
6. vstinner commented on Mar 10, 2024

   

   vstinner

   on Mar 10, 2024

   Member

   More actions

   I don't think that pythoncapi_compat.h design is a good fit to implement these features. I would be more comfortable with a library, rather than a header file.

   I created the experimental https://github.com/vstinner/deadparrot/ project which is a shared library. Maybe this project is a better home for PyType_FromMetaclass().
7. vstinner commented on Mar 10, 2024

   

   vstinner

   on Mar 10, 2024

   Member

   More actions

   @wjakob already has an implementation, see python/cpython#93012 (comment).

   I tried to implement this function in the deadparrot project: draft PR vstinner/deadparrot#2 which does crash...

   I tried this implementation on Python 3.11. I had a hard time to make it work :-( It doesn't set tp_base properly. I had issues with tp_name (not set properly). I hacked the code to set tp_base and tp_name, but then I got negative reference counts and memory corruptions :-( I'm not sure that it's sustainable to maintain such code :-(

   I also tried to backport PyType_FromMetaclass() from Python 3.13: I gave up after 500 lines of C code, it pulls way too many internals which would be too expensive to maintain, and it requires access to internals which are simply not accessible from the output, especially slots offsets.

   I close the pythoncapi-compat issue. If someone wants to write a PR for the deadparrot project, please go ahead :-)
8. vstinner

   closed this as completedon Mar 10, 2024

Issue actions