Description

ericsnowcurrently

opened on Jun 14, 2023

Issue body actions

Once we moved to per-interpreter GIL, the promises in the docs no longer hold:

...where the allocation must be performed with the GIL held.

It's still fine for pymalloc, but any custom, non-wrapping "mem"/"object" allocators would need to be updated to be thread-safe or per-interpreter.

I have a PR up that does an okay job of adapting such allocators: gh-105619. However, it penalizes use of such an allocator in subinterpreters that have their own GIL.

Honestly, I'm leaning toward documenting that such allocators must be thread-safe or per-interpreter. From what I understand, the documented guarantees (in the docs and in PEP 445) are more about representing what pymalloc needs than what custom allocators need.

Perhaps the biggest question is: what projects would be impacted? I haven't had a chance yet to search for projects that use custom mem/object allocators that aren't thread-safe. I suspect there aren't more than two or three.

Linked PRs

• gh-105766: Document that Custom Allocators Must Be Thread-Safe #107519
• [3.12] gh-105766: Document that Custom Allocators Must Be Thread-Safe (gh-107519) #107522
• gh-105766: Clarify About Custom Allocators in C-API Docs #109035

Activity

1. ericsnowcurrently

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-subinterpreters

   topic-C-API

   3.12only security fixesonly security fixes

   3.13only security fixesonly security fixes

   on Jun 14, 2023
2. github-project-automation

   added this to Subinterpreterson Jun 14, 2023
3. github-project-automation

   moved this to Todo in Subinterpreterson Jun 14, 2023
4. encukou commented on Jun 14, 2023

   

   encukou

   on Jun 14, 2023

   Member

   More actions

   Can we add a flag, so that e.g. PyMem_SetAllocator(PYMEM_DOMAIN_MEM | PyMEM_THREADSAFE, ...) would set PYMEM_DOMAIN_MEM without the wrappers?

   Reacted by Eric Snow
5. ericsnowcurrently commented on Jul 26, 2023

   

   ericsnowcurrently

   on Jul 26, 2023

   MemberAuthor

   More actions

   python-dev thread: https://discuss.python.org/t/what-to-do-about-custom-allocators-and-the-gil/30327
6. ericsnowcurrently commented on Jul 28, 2023

   

   ericsnowcurrently

   on Jul 28, 2023

   MemberAuthor

   More actions

   I'm going to add a note in the docs that custom allocators (that have their own state), including hooks/wrappers, must be thread-safe.

   We could also qualify that to only when subinterpreters are used, but I'm not sure that matters. If we did qualify it that way, it may also make sense to add the flag @encukou suggested above (and its inverse).
7. ericsnowcurrently

   added a commit that references this issue on Jul 31, 2023

   gh-105766: Document that Custom Allocators Must Be Thread-Safe (gh-10…

   db361a3
8. miss-islington

   added a commit that references this issue on Jul 31, 2023

   pythongh-105766: Document that Custom Allocators Must Be Thread-Safe (p…

   b03ad52
9. ericsnowcurrently

   added a commit that references this issue on Jul 31, 2023

   [3.12] gh-105766: Document that Custom Allocators Must Be Thread-Safe (…

   fc4532a
10. ericsnowcurrently

    closed this as completedon Jul 31, 2023
11. github-project-automation

    moved this from Todo to Done in Subinterpreterson Jul 31, 2023
12. ambv

    added 2 commits that reference this issue on Oct 21, 2026

    Fix CI: restore memoryinterface doc anchor (pythongh-105766)

    388cbac

    Fix CI: restore memoryinterface doc anchor (pythongh-105766)

    efb292c

Issue actions