Description

markshannon

opened on Aug 23, 2023

Issue body actions

The current GC is both inefficient and can have very long pause times.
The efficiency can be improved by tweaking thresholds or by using fewer generations (see #100403), but that does nothing to shorten pause times.

We should use an incremental collector, it can improve efficiency and hugely reduce maximum pause times.

See #100403 for more details on the general idea and the algorithm.

Linked PRs

• GH-108362: Incremental GC implementation #108038
• GH-108362: Set old space for objects in finalizer list before adding to old space #115084
• GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)" #115132
• GH-108362: Incremental GC implementation #116199
• GH-108362: Incremental GC implementation #116206
• gh-108362: Retarget incremental GC changes to 3.14 #125453
• [3.14] gh-108362: Retarget incremental GC changes to 3.14 (GH-125453) #136851

Activity

1. iritkatriel

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Nov 27, 2023
2. markshannon

   added a commit that references this issue on Feb 5, 2024

   GH-108362: Incremental GC implementation (GH-108038)

   36518e6
3. encukou commented on Feb 6, 2024

   

   encukou

   on Feb 6, 2024

   Member

   More actions

   Several stable buildbots fail on the new assertion:

   test_thread_safety (test.test_multiprocessing_forkserver.test_processes.WithProcessesTestFinalize.test_thread_safety) ... Assertion failed: (gc_list_is_empty(to) || gc_old_space(to_tail) == gc_old_space(from_tail)), function gc_list_merge, file gc.c, line 320.
   Fatal Python error: Aborted
   Current thread 0x0000000170977000 (most recent call first):
     Garbage-collecting
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/util.py", line 189 in __init__
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/_test_multiprocessing.py", line 4595 in __init__
     File "__init__", line ??? in __init__
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/_test_multiprocessing.py", line 4618 in make_finalizers
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1026 in run
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1089 in _bootstrap_inner
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1046 in _bootstrap
   Thread 0x000000016f96b000 (most recent call first):
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/util.py", line 214 in __call__
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/util.py", line 296 in _run_finalizers
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/_test_multiprocessing.py", line 4607 in run_finalizers
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1026 in run
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1089 in _bootstrap_inner
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1046 in _bootstrap
   Thread 0x00000001e2c35ec0 (most recent call first):
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/_test_multiprocessing.py", line 4631 in test_thread_safety
     File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/unittest/case.py", line 589 in _callTestMethod
     ...
   

   Do you have a quick fix that can be added today? If not, please revert first and then investigate.
   (I can help, but not today.)
4. markshannon commented on Feb 6, 2024

   

   markshannon

   on Feb 6, 2024

   MemberAuthor

   More actions

   I'll take a look. I can't reproduce the failure locally and it will be tricky to diagnose without a C backtrace.
5. markshannon commented on Feb 6, 2024

   

   markshannon

   on Feb 6, 2024

   MemberAuthor

   More actions

   @encukou
   I can't fix it now.
   I'll do a revert PR tomorrow, unless you beat me to it.
6. markshannon

   added a commit that references this issue on Feb 7, 2024

   Revert "pythonGH-108362: Incremental GC implementation (pythonGH-108038…

   f0dd4df
7. pablogsal

   added a commit that references this issue on Feb 7, 2024

   GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)…

   8a3c499
8. encukou commented on Feb 7, 2024

   

   encukou

   on Feb 7, 2024

   Member

   More actions

   Thank you for the revert.

   The bigmem buildbot also failed on the change. For next PR fix, please run the bigmem tests (./python -m test test_bigmem -v -M 20G or however much RAM you can spare) and/or the bigmem buildbot.
9. fsc-eriker

   added 2 commits that reference this issue on Feb 14, 2024

   pythonGH-108362: Incremental GC implementation (pythonGH-108038)

   d28be26

   pythonGH-108362: Revert "pythonGH-108362: Incremental GC implementati…

   0b95cb5
10. oraluben

    added a commit that references this issue on Mar 5, 2024

    Backport pythonGH-108362: Incremental GC implementation

    af62957
11. markshannon

    added a commit that references this issue on Mar 20, 2024

    GH-108362: Incremental Cycle GC (GH-116206)

    1530932
12. vstinner

    added a commit that references this issue on Mar 20, 2024

    pythonGH-108362: Incremental Cycle GC (pythonGH-116206)

    f4b2af9
13. gvanrossum

    added a commit that references this issue on Mar 23, 2024

    Revert "pythonGH-108362: Incremental Cycle GC (pythonGH-116206)"

    e6480f7
14. gvanrossum commented on Mar 23, 2024

    

    gvanrossum

    on Mar 23, 2024

    Member

    More actions

    @markshannon I have two buildbot failures crashing on an assert in gc_list_merge() that you touched:

            assert(gc_list_is_empty(to) ||
                gc_old_space(to_tail) == gc_old_space(from_tail));

    This is in https://buildbot.python.org/all/#/builders/223/builds/5581/steps/5/logs/stdio and also in https://buildbot.python.org/all/#/builders/469/builds/7667/steps/6/logs/stdio.
15. 6 remaining items
16. picnixz

    added

    type-featureA feature request or enhancementA feature request or enhancement

    on Oct 14, 2024
17. ZupoLlask commented on Jul 13, 2025

    

    ZupoLlask

    on Jul 13, 2025

    More actions

    Hi Devs!

    Is this feature dead? Is it still retargeted to version 3.14? 🙏
18. sergey-miryanov commented on Jul 13, 2025

    

    sergey-miryanov

    on Jul 13, 2025

    Contributor

    More actions

    This is implemented on 3.14.

    Reacted by Christopher Probst-Ranly and Tim Martin
19. hugovk

    added a commit that references this issue on Jul 20, 2025

    gh-108362: Retarget incremental GC changes to 3.14 (#125453)

    dda9d00
20. miss-islington

    added a commit that references this issue on Jul 20, 2025

    pythongh-108362: Retarget incremental GC changes to 3.14 (pythonGH-12…

    7922fa6
21. hugovk

    added a commit that references this issue on Jul 20, 2025

    [3.14] gh-108362: Retarget incremental GC changes to 3.14 (GH-125453) (…

    2cb3b53
22. taegyunkim

    added a commit that references this issue on Aug 4, 2025

    pythongh-108362: Retarget incremental GC changes to 3.14 (python#125453)

    5e71e72
23. Agent-Hellboy

    added a commit that references this issue on Aug 19, 2025

    pythongh-108362: Retarget incremental GC changes to 3.14 (python#125453)

    6a82e53
24. kumaraditya303

    added a commit that references this issue on Sep 9, 2025

    [3.14] pythongh-108362: Retarget incremental GC changes to 3.14 (pyth…

    6d02aca
25. adorilson commented on Dec 10, 2025

    

    adorilson

    on Dec 10, 2025

    Contributor

    More actions

    Could it be closed?

    Reacted by Hugo van Kemenade
26. hugovk

    closed this as completedon Dec 10, 2025
27. RonnyPfannschmidt

    added 4 commits that reference this issue on May 28, 2026

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    53167e5

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    5fbde2c

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    9e43244

    fix: guard FDCapture.snap() against prematurely closed tmpfile (pytes…

    594e4f5

Issue actions