Description

neonene

opened on Mar 22, 2024

Issue body actions

Feature or enhancement

Proposal:

This issue is available to keep track of PRs, following the heap type conversion completed at #114314.

TODO:

• [ ] Make free_list in malloc_closure.c per-module variables. rejected
• Fix intentional memory leaks of StgInfo during a finalization: comment.
• Enable Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED(DONE): comment.
  FUTURE?: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED after a compatible PyGILState_Ensure() is introduced, see also the links below.

Links to documents:

• libffi: a foreign function interface library
• libffi - GitHub

Links to previous discussion of this feature:

• Isolate Stdlib Extension Modules #103092
• Convert _ctypes static types (and the "meta class") to heap types #114314
• PyGILState_Ensure() for sub-interpreters:
  • [subinterpreters] Make the PyGILState API compatible with subinterpreters #59956
  • [subinterpreters] Make the PyGILState API compatible with multiple interpreters #55124
    • _ctypes patch (2011)

Linked PRs

• gh-117142: ctypes: Unify meta tp slot functions #117143
• gh-117142: Port _ctypes to multi-phase init #117181
• gh-117142: ctypes: Migrate global vars to module state #117189
• gh-117142: ctypes: Clean up globals-to-fix.tsv #117544
• gh-117142: ctypes: Migrate global closure freelist to thunk-type state #117874
• gh-117142: ctypes: Migrate global closure freelist to module state #117875
• gh-117142: ctypes: Fix memory leak of StgInfo #118139
• gh-117142: Slightly hacky fix for memory leak of StgInfo #119424
• [3.13] gh-117142: Slightly hacky fix for memory leak of StgInfo (GH-119424) #119468
• gh-117142: Support Importing ctypes in Isolated Interpreters #119991
• [3.13] gh-117142: Support Importing ctypes in Isolated Interpreters (gh-119991) #120008

Activity

1. neonene

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Mar 22, 2024
2. erlend-aasland

   added

   extension-modulesC modules in the Modules dirC modules in the Modules dir

   topic-subinterpreters

   on Mar 22, 2024
3. github-project-automation

   added this to Subinterpreterson Mar 22, 2024
4. github-project-automation

   moved this to Todo in Subinterpreterson Mar 22, 2024
5. encukou

   added 2 commits that reference this issue on Mar 29, 2024

   gh-117142: ctypes: Migrate global vars to module state (GH-117189)

   7e2fef8

   gh-117142: ctypes: Unify meta tp slot functions (GH-117143)

   dd44ab9
6. NGRsoftlab

   added 2 commits that reference this issue on Apr 2, 2024

   Squashed commit of the following:

   1b5fe4c

   Squashed commit of the following:

   a016a54
7. encukou

   added a commit that references this issue on Apr 10, 2024

   gh-117142: Port _ctypes to multi-phase init (GH-117181)

   ef41182
8. diegorusso

   added 3 commits that reference this issue on Apr 17, 2024

   pythongh-117142: ctypes: Migrate global vars to module state (pythonG…

   09bae85

   pythongh-117142: ctypes: Unify meta tp slot functions (pythonGH-117143)

   904ecd8

   pythongh-117142: Port _ctypes to multi-phase init (pythonGH-117181)

   89c8b7b
9. neonene

   changed the title [-]Isolate the _ctypes extension module[/-] [+]Convert _ctypes extension module to multi-phase init[/+] on May 8, 2024
10. neonene commented on May 8, 2024

    

    neonene

    on May 8, 2024

    ContributorAuthor

    More actions

    Could anyone add this open issue to the base issue's todo (port _ctypes branch) as a checked (finished) leaf? Thanks.

    Reacted by Petr Viktorin
11. 9 remaining items

12. ericsnowcurrently commented on May 30, 2024

    

    ericsnowcurrently

    on May 30, 2024

    Member

    More actions

    Unless I was looking at an old main HEAD, _ctypes is still has py_mod_multiple_interpreters set to Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED. Is there are reason we haven't set it to Py_MOD_PER_INTERPRETER_GIL_SUPPORTED?

    CC @neonene
13. ericsnowcurrently

    reopened this on May 30, 2024
14. neonene commented on May 30, 2024

    

    neonene

    on May 30, 2024

    ContributorAuthor

    More actions

    In my understanding of #117181 (comment), the closure_fcn function in callbacks.c needs a new compatible PyGILState_Ensure() faimily like #59956 (comment) to enable Py_MOD_PER_INTERPRETER_GIL_SUPPORTED?

    Issue #55124 in 2011 has an experimantal patch: https://bugs.python.org/file20417/gilstateinterp.patch

    Reacted by Eric Snow
15. encukou commented on May 31, 2024

    

    encukou

    on May 31, 2024

    Member

    More actions

    Also mmap-based closure allocation needs a mutex, and I don't know how thread-safe the libffi API is. There might be other things.
16. ericsnowcurrently commented on Jun 1, 2024

    

    ericsnowcurrently

    on Jun 1, 2024

    Member

    More actions

    FYI, PyGILState_Ensure() should work fine with subinterpreters, as of 3.12. It was necessary to fix that for per-interpreter GIL.
17. ericsnowcurrently commented on Jun 1, 2024

    

    ericsnowcurrently

    on Jun 1, 2024

    Member

    More actions

    Regarding mmap and libffi, aren't those resolved for free-threading builds?
18. ericsnowcurrently

    added a commit that references this issue on Jun 3, 2024

    gh-117142: Support Importing ctypes in Isolated Interpreters (gh-119991)

    dba7a16
19. miss-islington

    added a commit that references this issue on Jun 3, 2024

    pythongh-117142: Support Importing ctypes in Isolated Interpreters (p…

    2ce8e0e
20. mliezun

    added a commit that references this issue on Jun 3, 2024

    pythongh-117142: Support Importing ctypes in Isolated Interpreters (p…

    a8c45a8
21. ericsnowcurrently

    added a commit that references this issue on Jun 3, 2024

    [3.13] gh-117142: Support Importing ctypes in Isolated Interpreters (g…

    c348e27
22. ericsnowcurrently

    closed this as completedon Jun 3, 2024
23. barneygale

    added a commit that references this issue on Jun 5, 2024

    pythongh-117142: Support Importing ctypes in Isolated Interpreters (p…

    e8ee15c
24. estyxx

    added 3 commits that reference this issue on Jul 17, 2024

    pythongh-117142: ctypes: Clean up c-analyzer .tsv files (pythonGH-117544

    91a674e

    pythongh-117142: Slightly hacky fix for memory leak of StgInfo (pytho…

    afde92e

    pythongh-117142: Support Importing ctypes in Isolated Interpreters (p…

    d136038

Issue actions