Description

colesbury

opened on Nov 29, 2023

Issue body actions

Feature or enhancement

Python's cyclic garbage collector relies on the global interpreter lock for thread-safety. There are a number of changes needed to make the GC thread-safe in --disable-gil builds.

My intention is to implement these as a series of smaller changes.

• Garbage collection is guarded by gcstate->collecting. This need to be made thread-safe.
• The --disable-gil builds should stop-the-world when finding garbage, but not when calling finalizers or destructors. (depends on Implement stop-the-world functionality (for --disable-gil builds) #111964)
• The --disable-gil builds should find GC-enabled objects by traversing the mimalloc heaps, because the _gc_next/_gc_prev lists are not thread-safe. (Note it's safe to use the lists during stop-the-world pauses; we just can't maintain them safely during normal Python execution).
• The --disable-gil builds should probably use only a single generation to avoid frequent stop-the-world pauses
• Eventually, we can get rid of the GC pre-header in --disable-gil builds to save memory. (This will require updating the trashcan mechanism.)

Remaining work:

• Fix scheduling of free-threaded GC (currently scheduled as if there is a young generation)
• Update Python docs
• Refactor out common code from gc.c and gc_free_threaded.c

See also #111964

Linked PRs

• gh-112529: Use atomic operations for gcstate->collecting #112533
• gh-112529: Track if debug allocator is used as underlying allocator #113747
• gh-112529: Use GC heaps for GC allocations in free-threaded builds #114157
• gh-112529: Implement GC for free-threaded builds #114262
• gh-112529: Remove PyGC_Head from object pre-header in free-threaded build #114564
• gh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFinalized #114732
• gh-112529: Stop the world around gc.get_referents #114823
• gh-112529: Make the GC scheduling thread-safe #114880
• gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488
• gh-112529: Revert "Use _PyThread_Id() in mimalloc in free-threaded build" #115524
• gh-112529: Don't untrack tuples or dicts with zero refcount #117370

Activity

1. colesbury

   added

   type-featureA feature request or enhancementA feature request or enhancement

   3.13only security fixesonly security fixes

   topic-free-threading

   on Nov 29, 2023
2. colesbury

   self-assigned this

   on Nov 29, 2023
3. colesbury

   added a commit that references this issue on Nov 29, 2023

   pythongh-112529: Use atomic operations for `gcstate->collecting`

   a88fd5e
4. pablogsal

   added a commit that references this issue on Dec 11, 2023

   gh-112529: Use atomic operations for `gcstate->collecting` (#112533)

   d70e27f
5. colesbury

   added 6 commits that reference this issue on Jan 5, 2024

   Merge branch 'pythongh-113688' into pythongh-112529-gc

   68af10c

   pythongh-112529: Track if debug allocator is used as underlying alloc…

   4734274

   pythongh-112529: Track if debug allocator is used as underlying alloc…

   9996c9b

   pythongh-112529: Use GC mimalloc heaps for GC objects

   4891c36

   pythongh-112529: Track if debug allocator is used as underlying alloc…

   223bf94

   pythongh-112529: Use GC mimalloc heaps for GC objects

   4b4ec14
6. colesbury

   added 2 commits that reference this issue on Jan 16, 2024

   pythongh-112529: Track if debug allocator is used as underlying alloc…

   491237f

   pythongh-112529: Implement GC for free-threaded builds

   8314c7c
7. DinoV

   added a commit that references this issue on Jan 16, 2024

   gh-112529: Track if debug allocator is used as underlying allocator (#…

   b331381
8. 36 remaining items
9. adorilson

   added a commit that references this issue on Mar 25, 2024

   pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…

   7c6c093
10. colesbury

    added 2 commits that reference this issue on Mar 29, 2024

    pythongh-112529: Don't untrack tuples or dicts with zero refcount

    d3ed5d7

    gh-112529: Don't untrack tuples or dicts with zero refcount (#117370)

    f05fb2e
11. NGRsoftlab

    added 2 commits that reference this issue on Apr 2, 2024

    Squashed commit of the following:

    1b5fe4c

    Squashed commit of the following:

    a016a54
12. diegorusso

    added 3 commits that reference this issue on Apr 17, 2024

    pythongh-112529: Make the GC scheduling thread-safe (python#114880)

    52df553

    pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…

    15c69a6

    pythongh-112529: Don't untrack tuples or dicts with zero refcount (py…

    e46e06b
13. colesbury commented on May 7, 2024

    

    colesbury

    on May 7, 2024

    ContributorAuthor

    More actions

    I'm going to close this as the free-threaded GC is implemented now. We can open a new issue for refactoring out the common code.
14. colesbury

    closed this as completedon May 7, 2024
15. Glyphack

    added 4 commits that reference this issue on Sep 2, 2024

    pythongh-112529: Use atomic operations for `gcstate->collecting` (pyt…

    8ae3ca8

    pythongh-112529: Track if debug allocator is used as underlying alloc…

    baba4a4

    pythongh-112529: Use GC heaps for GC allocations in free-threaded bui…

    cf6875c

    pythongh-112529: Implement GC for free-threaded builds (python#114262)

    42f2abe
16. LukasWoodtli

    added a commit that references this issue on Jan 22, 2025

    pythongh-112529: Make the GC scheduling thread-safe (python#114880)

    df74dc2

Issue actions