GeezerVison is showing github.com — [subinterpreters] Static types incorrectly share some objects between interpreters · Issue #94673 · python/cpython
[subinterpreters] Static types incorrectly share some objects between interpreters #94673DescriptionWhile static types (PyTypeObject values) don't themselves ever change (once PyType_Ready() has run), they do hold mutable data. This means they cannot be safely shared between multiple interpreters without a common GIL (and without state leaking between).
Mutable data:
- the object header (e.g. refcount), as with all objects
- otherwise immutable objects:
- ob_type (
__class__)
- tp_base (
__base__) - set by PyType_Ready() if not set
- tp_bases (
__bases__) - always set by PyType_Ready()
- tp_mro (
__mro__) - always set by PyType_Ready()
- tp_dict (
__dict__) - set by PyType_Ready() if not set
- mutable containers:
- tp_subclasses (
__subclasses__)
- tp_weaklist
(See https://docs.python.org/3/c-api/typeobj.html#tp-slots.)
(Note that tp_cache is no longer used.)
For the object header, if PEP 683 (immortal objects) is accepted then we can make static types immortal.
For the otherwise immutable objects, we can make sure each is immortal and then we're good. Even tp_dict is fine since it gets hidden behind types.MappingProxyType. We'd also need to either make sure each contained item is immortal.
For tp_subclasses we will need a per-interpreter copy, and do the proper lookup in the __subclasses__ getter. The cache could be stored on PyInterpreterState or even as a dict in tp_subclasses.
For tp_weaklist it's a similar story as for tp_subclasses. Note that tp_weaklist isn't very important for static types since they are never deallocated.
(The above is also discussed in PEP 684.)
CC @kumaraditya303
Linked PRs
Activityericsnowcurrently commented on Jul 9, 2022 kumaraditya303 commented on Jul 10, 2022 The current _PyInterpreterState_GET (which would be used here) static inline function is quite inefficient as there is a lot indirection and involves atomic reads to get the "current" thread and then get the interpreter state. To improve this we can use thread local 1 variables on supported platform such as Linux, Windows and macOS. On Linux with glibc, the thread local implementation uses fs segment register and on x86-64 it is very efficient compared to the current implementation.
This isn't required here but would be nice to investigate separately.
encukou commented on Jul 11, 2022
This means they cannot be safely shared between multiple interpreters without a common GIL
Python currently has a common GIL.
Shouldn't this be part of a PEP, rather than an issue? ericsnowcurrently commented on Jul 19, 2022 I have the changes ready for this:
added a commit that references this issue on Apr 12, 2024gh-94673: Clarify About Runtime State Related to Static Builtin Types (… gvanrossum commented on Apr 15, 2024 We now get a warning when building without pydebug:
Objects/typeobject.c:120:1: warning: unused function 'static_builtin_index_is_set' [-Wunused-function]
static_builtin_index_is_set(PyTypeObject *self)
^
1 warning generated.
ericsnowcurrently commented on Apr 16, 2024
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|