Description

erlend-aasland

opened on Feb 11, 2023

Issue body actions

Isolate the _io extension module by moving all global variables to module state, porting static types to heap types, and implementing multi-phase init. All global variables in the _io module are static types:

• Modules/_io/bufferedio.c: PyBufferedIOBase_Type
• Modules/_io/bufferedio.c: PyBufferedRWPair_Type
• Modules/_io/bufferedio.c: PyBufferedRandom_Type
• Modules/_io/bufferedio.c: PyBufferedReader_Type
• Modules/_io/bufferedio.c: PyBufferedWriter_Type
• Modules/_io/bytesio.c: PyBytesIO_Type
• Modules/_io/bytesio.c: _PyBytesIOBuffer_Type
• Modules/_io/fileio.c: PyFileIO_Type
• Modules/_io/iobase.c: PyIOBase_Type
• Modules/_io/iobase.c: PyRawIOBase_Type
• Modules/_io/textio.c: PyIncrementalNewlineDecoder_Type
• Modules/_io/textio.c: PyTextIOBase_Type
• Modules/_io/textio.c: PyTextIOWrapper_Type
• Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type

Converting the static types to heap types involves applying PEP-687 to _io.
Adapting multi-phase init involves applying PEP-489 to _io.

Linked PRs

• gh-101819: Remove _PyWindowsConsoleIO_Type from the Windows DLL #101904
• gh-101819: Remove _testcapi dependencies on specific _io symbols #101918
• gh-101819: Isolate _io #101948
• gh-101819: Adapt _io types to heap types, batch 1 #101949
• gh-101819: Prepare to modernize the _io extension #104178
• gh-101819: Adapt _io.PyWindowsConsoleIO_Type to heap type #104197
• gh-101819: Remove unused 'locale_module' from _io state #104246
• gh-101819: Port _io.PyIncrementalNewlineDecoder_Type to heap type #104249
• gh-101819: Port _io.PyBytesIOBuffer_Type to heap type #104264
• gh-101819: Refactor _io in preparation for module isolation #104334
• gh-101819: Harden _io init #104352
• gh-101819: Clean up _io windows console io after gh-104197 #104354
• gh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic #104355
• gh-101819: Adapt _io._Buffered* methods to Argument Clinic #104367
• gh-101819: Refactor _io futher in preparation for module isolation #104369
• gh-101819: Explicitly disallow pickle protocols 0 and 1 in _io #104370
• gh-101819: Adapt _io.TextIOBase methods to Argument Clinic #104383
• gh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic #104384
• gh-101819: Prepare _io._IOBase for module state #104386
• gh-101819: Fix inverted debug preprocessor check in winconsoleio.c #104388
• gh-101819: Fix _io clinic input for unused base class method stubs #104418

Reacted by Kumar Aditya

Activity

1. erlend-aasland

   added

   type-featureA feature request or enhancementA feature request or enhancement

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   topic-subinterpreters

   on Feb 11, 2023
2. erlend-aasland commented on Feb 11, 2023

   

   erlend-aasland

   on Feb 11, 2023

   ContributorAuthor

   More actions

   cc. @kumaraditya303
3. ezio-melotti

   added this to Subinterpreterson Feb 11, 2023
4. kumaraditya303

   assigned

   erlend-aasland

   and

   kumaraditya303

   on Feb 11, 2023
5. kumaraditya303

   added

   3.12only security fixesonly security fixes

   on Feb 11, 2023
6. erlend-aasland commented on Feb 13, 2023

   

   erlend-aasland

   on Feb 13, 2023

   ContributorAuthor

   More actions

   I see that _PyWindowsConsoleIO_Type is marked with PyAPI_DATA, but I can't find it in Misc/stable_abi.toml. @zooba or @eryksun, do you know why this is so? I did a quick code search on GitHub, but I only got hits in various clones of the CPython code base.

   cpython/Modules/_io/_iomodule.h

   Lines 24 to 30 in 2db2c4b

   	#ifndef Py_LIMITED_API
   	#ifdef MS_WINDOWS
   	extern PyTypeObject PyWindowsConsoleIO_Type;
   	PyAPI_DATA(PyObject *) _PyWindowsConsoleIO_Type;
   	#define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), (PyTypeObject*)_PyWindowsConsoleIO_Type))
   	#endif /* MS_WINDOWS */
   	#endif /* Py_LIMITED_API */

   UPDATE: I see it was added by Steve in 312cef7, probably because of the PyWindowsConsoleIO_Check checks in PC/_testconsole.c and Python/pylifecycle.c:

   cpython/PC/_testconsole.c

   Lines 54 to 57 in 2db2c4b

   	if (!PyWindowsConsoleIO_Check(file)) {
   	PyErr_SetString(PyExc_TypeError, "expected raw console object");
   	return NULL;
   	}

   cpython/Python/pylifecycle.c

   Lines 58 to 61 in 2db2c4b

   	extern PyTypeObject PyWindowsConsoleIO_Type;
   	# define PyWindowsConsoleIO_Check(op) \
   	(PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type))
   	#endif

   cpython/Python/pylifecycle.c

   Lines 2359 to 2363 in 2db2c4b

   	#ifdef MS_WINDOWS
   	/* Windows console IO is always UTF-8 encoded */
   	if (PyWindowsConsoleIO_Check(raw))
   	encoding = L"utf-8";
   	#endif

   Perhaps exposing _PyWindowsConsoleIO_Type through a CPython internal capsulated C API could be an option?
7. erlend-aasland commented on Feb 13, 2023

   

   erlend-aasland

   on Feb 13, 2023

   ContributorAuthor

   More actions

   Other issues that must be resolved before continuing: _PyBytesIOBuffer_Type is used by _testcapimodule. Since that type is not exposed in _io, we cannot simply use PyImport_Import and look it up. One possibility is to expose it through a CPython internal capsule C API.
8. erlend-aasland commented on Feb 13, 2023

   

   erlend-aasland

   on Feb 13, 2023

   ContributorAuthor

   More actions

   ... or we could simply expose the types mentioned in #101819 (comment) and #101819 (comment) in the _io module.
9. zooba commented on Feb 13, 2023

   

   zooba

   on Feb 13, 2023

   Member

   More actions

   Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

   Reacted by Erlend E. Aasland
10. encukou commented on Feb 13, 2023

    

    encukou

    on Feb 13, 2023

    Member

    More actions

    The type is explicitly not part of the limited API/stable ABI, see #ifndef Py_LIMITED_API.

    Reacted by Erlend E. Aasland
11. erlend-aasland commented on Feb 13, 2023

    

    erlend-aasland

    on Feb 13, 2023

    ContributorAuthor

    More actions

    Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

    IIUC, we can remove this from the DLL and instead export it via _io and look it up from the module using the C API (PyImport_Import, etc.).
12. zooba commented on Feb 13, 2023

    

    zooba

    on Feb 13, 2023

    Member

    More actions

    Yeah, if it's got a leading underscore then we can do whatever we need to make our tests work.

    Reacted by Erlend E. Aasland

    Reacted by Erlend E. Aasland
13. 50 remaining items
14. erlend-aasland

    added a commit that references this issue on May 11, 2023

    gh-101819: Fix inverted debug preprocessor check in winconsoleio.c (#…

    ddc0e70
15. carljm

    added a commit that references this issue on May 11, 2023

    Merge branch 'main' into supermon

    fb47955
16. erlend-aasland

    added 2 commits that reference this issue on May 12, 2023

    gh-101819: Prepare _io._IOBase for module state (#104386)

    15795b5

    pythongh-101819: Fix _io clinic input for unused base class method stubs

    b5e7d6e
17. carljm

    added a commit that references this issue on May 12, 2023

    Merge branch 'main' into supermon

    8efd7c6
18. kumaraditya303

    added a commit that references this issue on May 15, 2023

    gh-101819: Isolate `_io` (#101948)

    186bf39
19. erlend-aasland

    added a commit that references this issue on May 15, 2023

    gh-101819: Fix _io clinic input for unused base class method stubs (#…

    b378d99
20. erlend-aasland

    closed this as completedon May 15, 2023
21. github-project-automation

    moved this to Done in Subinterpreterson May 15, 2023
22. carljm

    added 2 commits that reference this issue on May 15, 2023

    Merge branch 'main' into supermon

    343867a

    Merge branch 'main' into cached_property_docs

    452d9f5
23. vstinner commented on May 15, 2023

    

    vstinner

    on May 15, 2023

    Member

    More actions

    @erlend-aasland erlend-aasland closed this as completed

    Great achievement!

    Reacted by Erlend E. Aasland
24. serhiy-storchaka commented on Oct 23, 2023

    

    serhiy-storchaka

    on Oct 23, 2023

    Member

    More actions

    It causes a crash during garbage collection. See #111049, #111211.
25. python-sidebar

    added 2 commits that reference this issue on Sep 1, 2024

    pythongh-101819: Adapt _io types to heap types, batch 1 (pythonGH-101949

    34e4ff2

    pythongh-101819: Adapt _io types to heap types, batch 1 (pythonGH-101949

    575d8d0

Issue actions