GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [3.13] GH-124567: Revert the Incremental GC in 3.13 by Yhg1s · Pull Request #124770 · python/cpython

Conversation

@Yhg1s

@Yhg1s Yhg1s commented Sep 29, 2024 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Revert the Incremental GC in 3.13.

This reverts PRs #116206 (commit 1530932), #117120 (e28477f), #117213 (8bef34f), #117422 (ddf814d), #118313 (2ba1aed), #123268 (b1372e2) and #123395 (aca6511). Does not revert the raising of the first generation threshold from 700 to 2000, nor changes to C APIs (even private ones).

  • Issue: Incremental GC causes a significant slowdown for Sphinx #124567

📚 Documentation preview 📚: https://cpython-previews--124770.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead and Privat33r-dev reacted with heart emoji
All reactions
  • ❤️ 2 reactions
Yhg1s added 7 commits September 29, 2024 20:45
@Yhg1s
Revert "[3.13] pythonGH-117759: Document incremental GC (pythonGH-123266
95fe67e
) (python#123395)"

This reverts commit aca6511.
@Yhg1s
Revert "[3.13] pythonGH-122298: Restore printing of GC stats (pythonG…
8e9333e
…H-123261) (python#123268)"

This reverts commit b1372e2.
@Yhg1s
Revert "pythongh-117657: TSAN fix race on gstate->young.count (pyth…
3155e44
…on#118313)"

This reverts commit 2ba1aed.
@Yhg1s
Revert "Silence compiler warnings in gc.c (python#117422)"
8f60f05
This reverts commit ddf814d.
@Yhg1s
Revert "pythonGH-117108: Set the "old space bit" to "visited" for all…
2992289
… young objects (python#117213)"

This reverts commit 8bef34f.
@Yhg1s
Revert "pythonGH-117108: Change the size of the GC increment to about…
60a248b
… 1% of the total heap size. (pythonGH-117120)"

This reverts commit e28477f.
@Yhg1s
Revert "pythonGH-108362: Incremental Cycle GC (pythonGH-116206)"
d15eace
This reverts commit 1530932.
@Yhg1s
Yhg1s requested review from ericsnowcurrently, markshannon and pablogsal as code owners September 29, 2024 21:16
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 29, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Sep 29, 2024
Incremental GC causes a significant slowdown for Sphinx #124567
Closed
@Yhg1s
Yhg1s requested a review from nascheme September 29, 2024 21:17
@Yhg1s

Yhg1s commented Sep 29, 2024

Copy link
Copy Markdown
Member Author

This is a fairly simple revert, we'll have to address the what's new docs and news entries but I want to get this out there for review and testing in the meantime.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s added 3 commits September 29, 2024 23:21
@Yhg1s
Fix merge snafu in What's New.
ffcd513
@Yhg1s
Fix the free-threaded build.
03caeb5
@Yhg1s
Undo a few revers where there is no harm in the original versions.
cff0436
@AlexWaygood

AlexWaygood commented Sep 29, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Happy to report that this cuts around 50% off the time it takes to do an HTML build of the CPython docs locally with Sphinx (build options were --with-lto=fat --enable-optimizations). On the tip of the 3.13 branch, the second script I posted in #124567 (comment) reports a Sphinx docs-build time of 66.28s for me locally; with this PR, it reports 34.38s.

hauntsaninja, Privat33r-dev, and gpshead reacted with thumbs up emoji
All reactions
  • 👍 3 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nascheme

nascheme commented Sep 29, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

A little benchmarking from my desktop PC. "Sphinx" is build/html for typing.rst. The debug column is for the --with-debug build. The opt column is for optimized build without LTO or PGO. The "mypy" runs are for the mypy testsuite after disabling the gc.set_threshold() call it does (it sets a much higher threshold than default one). The mypy run creates quite a big heap but not much cyclic garbage. The Sphinx run is a smaller heap but quite a lot of cyclic garbage with smallish objects.

Based on this limited testing, it's looking pretty good.

Description debug [sec] opt [sec] peak rss [MB]
Sphinx, 3.12 5.17 1.71 88.1
Sphinx, 3.13 inc 10.25 1.92 96.9
Sphinx, 3.13 revert 6.03 1.87 96.9
Mypy, 3.12   25.54 274.4
Mypy, 3.13 inc   28.28 276.2
Mypy, 3.13 revert   25.51 272.4
hauntsaninja reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s
Fix TSan build of freethreaded python.
a0d5270
AlexWaygood
AlexWaygood reviewed Sep 29, 2024
View reviewed changes
Comment thread Modules/gcmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@AlexWaygood

AlexWaygood commented Sep 29, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

We can also see that the Doctest CI job on this PR took only 4 minutes, as opposed to e.g. 16-17 minutes on other recent 3.13 PRs such as

  • https://github.com/python/cpython/actions/runs/11087447714/job/30806712091?pr=124738
  • https://github.com/python/cpython/actions/runs/11092139738/job/30816812499?pr=124764
  • https://github.com/python/cpython/actions/runs/11084048667/job/30798940333?pr=124727
Privat33r-dev reacted with thumbs up emoji gpshead reacted with hooray emoji
All reactions
  • 👍 1 reaction
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s
Reinstate cast.
1ba3555
nascheme
nascheme approved these changes Sep 30, 2024
View reviewed changes

@nascheme nascheme left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I compared gc.c from before GH-108362 (d5ebf8b) and it looks like your reversion got the right bits.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 30, 2024
@Yhg1s

Yhg1s commented Sep 30, 2024

Copy link
Copy Markdown
Member Author

I believe the ABI changes are acceptable (the size of a private struct within another private struct changes, so later fields in the outer struct move around -- but as far as I can tell those structs are not exposed to users). The hypothesis failure seems unrelated (but possibly worth investigating).

AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

markshannon
markshannon approved these changes Sep 30, 2024
View reviewed changes

@markshannon markshannon left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One (unimportant) whitespace regression, otherwise this looks correct to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/library/gc.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Sep 30, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

I believe the ABI changes are acceptable (the size of a private struct within another private struct changes, so later fields in the outer struct move around -- but as far as I can tell those structs are not exposed to users). The hypothesis failure seems unrelated (but possibly worth investigating).

I checked the ABI changes and they are safe as this just complains about the size change of _gc_runtime_state. This won't affect wheels or anything using the ABI and for profilers the offsets have changed but this is not a problem if they are using the new offsets fields here:

cpython/Include/internal/pycore_runtime.h

Line 193 in fe58596

uint64_t collecting;

This may affect still some profilers that are vendoring headers like Austin (https://github.com/P403n1x87/austin/blob/4aead107f10e55a834b1f64a46ebc97fdb70f238/src/python/gc.h) but that is impossible to avoid (also here we were lucky because they haven't released wheels for 3.13 yet!).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pablogsal
pablogsal approved these changes Sep 30, 2024
View reviewed changes
@AA-Turner

AA-Turner commented Sep 30, 2024

Copy link
Copy Markdown
Member

AA-Turner@d7acbbd resolves What's New by just removing all references to the incremental GC.

I'm not sure there's much value discussing that the incremental GC was in pre-releases and was then reverted in What's New as it will serve to confuse the majority of readers, the detail should be in the NEWS entry.

A

AlexWaygood, gpshead, and Privat33r-dev reacted with thumbs up emoji
All reactions
  • 👍 3 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead
gpshead reviewed Sep 30, 2024
View reviewed changes
Comment thread Doc/whatsnew/3.13.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s added 2 commits September 30, 2024 22:39
@Yhg1s
Update the ABI.
f95093d
@Yhg1s
Incorporate AA Turner's doc update
6947c05
(AA-Turner@d7acbbd)
@Yhg1s
Yhg1s requested a review from encukou as a code owner September 30, 2024 20:50
Yhg1s and others added 2 commits September 30, 2024 13:51
@Yhg1s
Merge branch '3.13' into 3.13-revert-incremental-gc
f4b14d5
@blurb-it
📜🤖 Added by blurb_it.
8504cc0
@Yhg1s
Yhg1s enabled auto-merge (squash) September 30, 2024 21:07
@Yhg1s
Yhg1s merged commit e0eb44a into python:3.13 Sep 30, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Sep 30, 2024
@Privat33r-dev Privat33r-dev mentioned this pull request Oct 1, 2024
Doc tests action takes too long #118891
Closed
@ned-deily

ned-deily commented Oct 1, 2024

Copy link
Copy Markdown
Member

FWIW, a negative performance impact of the revert to some microbenchmarks: #124567 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner AA-Turner mentioned this pull request Oct 3, 2024
GH-109975: Copyedit 3.13 What's New: Remove references to the incremental GC #124947
Merged
@hugovk hugovk mentioned this pull request Oct 14, 2024
gh-108362: Retarget incremental GC changes to 3.14 #125453
Merged
hugovk pushed a commit to hugovk/cpython that referenced this pull request Nov 1, 2024
@Yhg1s @AA-Turner @hugovk
[3.13] pythonGH-124567: Revert the Incremental GC in 3.13 (python#124770
ebaca55
)

Revert the incremental GC in 3.13, since it's not clear that without further turning, the benefits outweigh the costs.

Co-authored-by: Adam Turner <[email protected]>
hugovk added a commit to hugovk/cpython that referenced this pull request Nov 4, 2024
@Yhg1s @AA-Turner @hugovk
pythonGH-124567: Revert the Incremental GC in 3.14 (python#124770)
a812a7a
Revert the incremental GC in 3.14, since it's not clear that without further turning, the benefits outweigh the costs.

Co-authored-by: Adam Turner <[email protected]>
Co-authored-by: Hugo van Kemenade <[email protected]>
hugovk added a commit to hugovk/cpython that referenced this pull request Nov 11, 2024
@Yhg1s @AA-Turner @hugovk
pythonGH-124567: Revert the Incremental GC in 3.14 (python#124770)
2cd7214
Revert the incremental GC in 3.14, since it's not clear that without further turning, the benefits outweigh the costs.

Co-authored-by: Adam Turner <[email protected]>
Co-authored-by: Hugo van Kemenade <[email protected]>
@Yhg1s
Yhg1s deleted the 3.13-revert-incremental-gc branch August 13, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@AlexWaygood AlexWaygood AlexWaygood left review comments
@nascheme nascheme nascheme approved these changes
@markshannon markshannon markshannon approved these changes
@pablogsal pablogsal pablogsal approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@encukou encukou Awaiting requested review from encukou encukou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@Yhg1s @AlexWaygood @nascheme @pablogsal @AA-Turner @ned-deily @gpshead @markshannon
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.