GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — Update GC docs for incremental collection. by markshannon · Pull Request #1379 · python/devguide Conversation
|
|
|
||
|
|
||
| To collect all unreachable cycles in the heap, the garbage collector must scan the | ||
| whole heap. This whole heap scan is called a cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe consider a different term here (and throughout below) since "cycle" already has an important meaning here and it can be confusing if we overload it:
| whole heap. This whole heap scan is called a cycle. | |
| whole heap. This whole heap scan is called a full collection. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe instead of cycle use "full scavenge".
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A collection is an increment, so that's not a good term.
Cycle is overloaded, so that's not great either.
"Scavenge" is the least ambiguous, but obscure.
Anyone have any other suggestions? I'll go with "scavenge" if not.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| * All any objects reachable from those objects that have not yet been scanned this cycle. | ||
|
|
||
| Any objects surviving this collection are moved to the old generation. | ||
| ollection from cycles. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ollection from cycles. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| * The oldest fraction of the old generation | ||
| * All any objects reachable from those objects that have not yet been scanned this cycle. | ||
|
|
||
| Any objects surviving this collection are moved to the old generation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe clarify that the objects that started in the old generation are considered "youngest of the old" instead of "oldest of the old" now (there's probably a better way of phrasing it).
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It isn't really the oldest, it is the "least recently scanned". I'll rework this section.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| implementation for the default build uses incremental collection with two | ||
| generations. | ||
|
|
||
| The purpose of generations is to take advantage of what is known as the weak |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| The purpose of generations is to take advantage of what is known as the weak | |
| Generational garbage collection takes advantage of what is known as the weak |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| two generations: young and old. Every new object starts in the young generation. | ||
|
|
||
|
|
||
| To collect all unreachable cycles in the heap, the garbage collector must scan the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps add detection as distinct from collection.
| To collect all unreachable cycles in the heap, the garbage collector must scan the | |
| To detect and collect all unreachable cycles in the heap, the garbage collector must scan the |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| To collect all unreachable cycles in the heap, the garbage collector must scan the | ||
| whole heap. This whole heap scan is called a cycle. | ||
|
|
||
| In order to limit the time each garbage collection takes, the previous algorithm |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| In order to limit the time each garbage collection takes, the previous algorithm | |
| To limit the time each garbage collection takes, the detection and collection algorithm |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
|
||
| * The young generation | ||
| * The oldest fraction of the old generation | ||
| * All any objects reachable from those objects that have not yet been scanned this cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * All any objects reachable from those objects that have not yet been scanned this cycle. | |
| * All objects reachable from those objects that have not yet been scanned this cycle. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
|
||
|
|
||
| To collect all unreachable cycles in the heap, the garbage collector must scan the | ||
| whole heap. This whole heap scan is called a cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe instead of cycle use "full scavenge".
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
Uh oh!
There was an error while loading. Please reload this page.
| When all objects in the heap have been scanned a cycle ends, and all objects are | ||
| considered unscanned again. | ||
|
|
||
| In order to collect all unreachable cycles, each increment must contain all of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To make sure that I understand this section on unreachable cycles, I think you're saying that we want to ensure that we fully capture the unreachable cycle because we want to ensure that the cycle is either fully gc'd or not, to avoid partial processing, which could be problematic later on?
If so, maybe it's worth mentioning explicitly.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't scan the full cycle at once, we cannot collect it. It is otherwise safe.
The old generational collector would scan part cycles all the time; it just delayed the collection of the cycle, at worst until a full collection.
With the incremental collector, if we only scan part of a cycle, it may never be collected. Which would be a problem.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
Thanks for the comments and suggestions. |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
| Survivors are moved to the back of the scanned list. The old part of increment is taken | ||
| from the front of the unscanned list. | ||
|
|
||
| When a cycle starts, no objects in the heap are considered to have been scanned. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using cycle of a "reference cycle" and a "gc execution" it's a bit confusing, can we use other terminology?
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've used "full scavenge"
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| In order to make sure that the whole of any unreachable cycle is contained in an | ||
| increment, all unscanned objects reachable from any object in the increment must | ||
| be included in the increment. | ||
| Thus, to form a complete increment we perform a transitive closure over reachable, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As much as I enjoy proper terminology myself. Many people reading this document may not immediately know what a transitive closure is, could you add a brief explanation in parentheses or just explain the process?
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
|
||
| * The young generation | ||
| * The least recently scanned fraction of the old generation. | ||
| * All objects reachable from those objects that have not yet been scanned this cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as my other comments about "cycle"
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| unscanned objects from the initial increment. | ||
| We can exclude scanned objects, as they must have been reachable when scanned. | ||
| If a scanned object becomes part of an unreachable cycle after being scanned, it | ||
| will not be collected this cycle, but it will be collected next cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can benefit from a concrete simple example showing how this can happen and how will be eventually cleaned. Either in English or with some diagram or pseudo code
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| generations. Every collection operates on the entire heap. | ||
| two generations: young and old. Every new object starts in the young generation. | ||
|
|
||
| To detect and collect all unreachable cycles in the heap, the garbage collector |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps use objects instead of cycles here.
| To detect and collect all unreachable cycles in the heap, the garbage collector | |
| To detect and collect all unreachable objects in the heap, the garbage collector |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| of three parts: | ||
|
|
||
| * The young generation | ||
| * The least recently scanned fraction of the old generation. |
willingc
Aug 28, 2024
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
edited
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * The least recently scanned fraction of the old generation. | |
| * The old generation's least recently scanned heap found in the scanned objects list |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
|
||
| * The young generation | ||
| * The least recently scanned fraction of the old generation. | ||
| * All objects reachable from those objects that have not yet been scanned this cycle. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| * All objects reachable from those objects that have not yet been scanned this cycle. | |
| * All objects reachable from those objects that have not yet been scanned. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| * The least recently scanned fraction of the old generation. | ||
| * All objects reachable from those objects that have not yet been scanned this cycle. | ||
|
|
||
| Any objects surviving this collection are moved to the old generation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Any objects surviving this collection are moved to the old generation. | |
| Any young generation objects surviving this collection are moved to the old generation, and reachable objects in the old generation remain in the old generation. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| * All objects reachable from those objects that have not yet been scanned this cycle. | ||
|
|
||
| Any objects surviving this collection are moved to the old generation. | ||
| The old generation is composed of two lists, scanned and unscanned. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| The old generation is composed of two lists, scanned and unscanned. | |
| The old generation is composed of two lists: scanned increments and unscanned. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
Uh oh!
There was an error while loading. Please reload this page.
|
Please note the inconsistency in argument or variable names: This devguide speaks of |
All reactions
-
👍 1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
| If a scanned object becomes part of an unreachable cycle after being scanned, it | ||
| will not be collected this cycle, but it will be collected next cycle. | ||
|
|
||
| The GC implementation for the free-threaded build does not use incremental collection. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This probably should be called out in an Important block.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
|
||
| In order to decide when to run, the collector keeps track of the number of object | ||
| allocations and deallocations since the last collection. When the number of | ||
| allocations minus the number of deallocations exceeds ``threshold_0``, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| allocations minus the number of deallocations exceeds ``threshold_0``, | |
| allocations minus the number of deallocations exceeds ``threshold0``, |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
| collection starts. ``threshold_1`` determines the fraction of the old | ||
| collection that is included in the increment. | ||
| The fraction is inversely proportional to ``threshold_1``, | ||
| as historically a larger ``threshold_1`` meant that old generation | ||
| collections were performed less frequency. | ||
| ``threshold2`` is ignored. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct as mentioned in @mr-bronson's message. Good catch @mr-bronson. 😄
| collection starts. ``threshold_1`` determines the fraction of the old | |
| collection that is included in the increment. | |
| The fraction is inversely proportional to ``threshold_1``, | |
| as historically a larger ``threshold_1`` meant that old generation | |
| collections were performed less frequency. | |
| ``threshold2`` is ignored. | |
| collection starts. ``threshold1`` determines the fraction of the old | |
| collection that is included in the increment. | |
| The fraction is inversely proportional to ``threshold1``, | |
| as historically a larger ``threshold1`` meant that old generation | |
| collections were performed less frequency. | |
| ``threshold2`` is ignored. |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
willingc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good to me! Thanks @markshannon.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
Let's hold until we figure out https://discuss.python.org/t/incremental-gc-and-pushing-back-the-3-13-0-release/65285 |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Garbage collection doc has been moved to the CPython repo. Unless there is an objection, I recommend closing this devguide issue. |
All reactions
-
👍 2 reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
| The content of these generations can be examined using the | ||
| ``gc.get_objects(generation=NUM)`` function and collections can be triggered | ||
| specifically in a generation by calling ``gc.collect(generation=NUM)``. | ||
| Prior to 3.13, there we three generations. For that reason the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Prior to 3.13, there we three generations. For that reason the | |
| Prior to 3.14, there we three generations. For that reason the |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
All these docs are now in main |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assignedLabels
Projects
Milestone
No milestoneDevelopment
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
7 participants
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.