GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-115103: Update gc.collect to process delayed objects by corona10 · Pull Request #116238 · python/cpython

Conversation

@corona10

@corona10 corona10 commented Mar 2, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Add delayed reclamation mechanism for free-threaded build (QSBR) #115103

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@corona10
pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed
c28c79f
@corona10
corona10 requested a review from colesbury March 2, 2024 07:24
@bedevere-app bedevere-app Bot mentioned this pull request Mar 2, 2024
Add delayed reclamation mechanism for free-threaded build (QSBR) #115103
Closed
@corona10 corona10 added skip news topic-free-threading labels Mar 2, 2024
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

!buildbot nogil
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 2, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit c28c79f 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • x86-64 MacOS Intel NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • AMD64 Ubuntu NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10
corona10 marked this pull request as ready for review March 2, 2024 07:31
@corona10
corona10 requested a review from pablogsal as a code owner March 2, 2024 07:31
@bedevere-app bedevere-app Bot added the awaiting core review label Mar 2, 2024
@corona10 corona10 mentioned this pull request Mar 2, 2024
gh-112087: Use QSBR technique for list_new/clear for free-thread build #115875
Merged
corona10
corona10 commented Mar 2, 2024
View reviewed changes
Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@corona10
Update Lib/test/support/__init__.py
81f333a
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

!buildbot nogil
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Mar 2, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 81f333a 🤖

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • x86-64 MacOS Intel ASAN NoGIL PR
  • x86-64 MacOS Intel NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Ubuntu NoGIL Refleaks PR
  • AMD64 Ubuntu NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

See #115875 (comment)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed Mar 2, 2024
View reviewed changes
Comment thread Modules/gcmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@corona10
Address code review
59b3fc4
corona10
corona10 commented Mar 2, 2024
View reviewed changes
Comment thread Modules/gcmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
corona10
corona10 commented Mar 2, 2024
View reviewed changes
Comment thread Modules/clinic/gcmodule.c.h Resolved

Uh oh!

There was an error while loading. Please reload this page.
@corona10
corona10 requested a review from pablogsal March 2, 2024 13:24
colesbury
colesbury reviewed Mar 2, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this as part of the normal gc.collect() instead of adding a separate call. We want gc.collect() to be the go-to way to ensuring that things that should be freed are freed -- we already empty freelists and merge refcounts and will probably have a few more similar tasks in the near future (like collecting shared dictionary keys).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Python/gc_free_threading.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Python/gc_free_threading.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
corona10 added 2 commits March 3, 2024 02:32
@corona10
Address code review
85b419c
@corona10
nit
a187b03
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

Yeah more better!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
colesbury
colesbury approved these changes Mar 2, 2024
View reviewed changes

@colesbury colesbury left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Mar 2, 2024
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

@pablogsal Do you have any comments?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed Mar 2, 2024
View reviewed changes
Comment thread Python/gc_free_threading.c
_PyEval_StopTheWorld(interp);
// merge refcounts for all queued objects
merge_all_queued_objects(interp, state);
process_delayed_frees(interp);

@pablogsal pablogsal Mar 2, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit/question: _PyEval_StopTheWorld also locks _PyRuntime. Maybe is worth moving the lock up?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@colesbury colesbury Mar 2, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_PyEval_StopTheWorld locks _PyRuntime temporarily to loop over threads, but does not hold the lock

We want process_delayed_frees after the stop-the-world call because it guarantees that we can free all the "delayed free" pointers. If we moved it up before the _PyEval_StopTheWorld there might be some memory blocks we don't free.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@pablogsal pablogsal Mar 2, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks for clarifying

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@pablogsal pablogsal Mar 2, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we moved it up before the _PyEval_StopTheWorld there might be some memory blocks we don't free.

With "moving the lock up" i meant to have a unique locking for both functions, not to move this call before the other. But your comment still applies

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
pablogsal
pablogsal approved these changes Mar 2, 2024
View reviewed changes
@corona10
corona10 merged commit 2e91578 into python:main Mar 2, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Mar 2, 2024
@corona10
corona10 deleted the gh-115103-refleak branch March 2, 2024 21:44
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

Nah I forgot the change the PR title :( anyway, I merged.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@corona10 corona10 changed the title gh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed gh-115103: Update gc.collect to process delayed objects Mar 2, 2024
@corona10

corona10 commented Mar 2, 2024

Copy link
Copy Markdown
Member Author

I left the comment about commit title for the future tracker: #115103 (comment)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
corona10 added a commit to corona10/cpython that referenced this pull request Mar 2, 2024
@corona10
Revert "pythongh-115103: Update refleak checker to trigger _PyMem_Pro…
0f19393 
…cessDelayed (pythongh-116238)"

This reverts commit 2e91578.
corona10 added a commit that referenced this pull request Mar 2, 2024
@corona10
gh-115103: Update gc.collect to process delayed objects (gh-116251)
d1fd060 
* Revert "gh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed (gh-116238)"

This reverts commit 2e91578.

* gh-115103: Update gc.collect to process delayed objects
@corona10

corona10 commented Mar 2, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I tried to fix the commit title for tracking with the double revert technique but it doesn't work for git blame: d1fd060

Let's forget about it :(
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024
@corona10 @woodruffw
pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDela…
68d5ed9 
…yed (pythongh-116238)
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024
@corona10 @woodruffw
pythongh-115103: Update gc.collect to process delayed objects (python…
167895b 
…gh-116251)

* Revert "pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed (pythongh-116238)"

This reverts commit 2e91578.

* pythongh-115103: Update gc.collect to process delayed objects
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
@corona10 @adorilson
pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDela…
dd06c33 
…yed (pythongh-116238)
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
@corona10 @adorilson
pythongh-115103: Update gc.collect to process delayed objects (python…
5c4a04d 
…gh-116251)

* Revert "pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed (pythongh-116238)"

This reverts commit 2e91578.

* pythongh-115103: Update gc.collect to process delayed objects
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@corona10 @diegorusso
pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDela…
4b11f2d 
…yed (pythongh-116238)
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@corona10 @diegorusso
pythongh-115103: Update gc.collect to process delayed objects (python…
dd1b844 
…gh-116251)

* Revert "pythongh-115103: Update refleak checker to trigger _PyMem_ProcessDelayed (pythongh-116238)"

This reverts commit 2e91578.

* pythongh-115103: Update gc.collect to process delayed objects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@colesbury colesbury colesbury approved these changes
@pablogsal pablogsal pablogsal approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@corona10 @bedevere-bot @colesbury @pablogsal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.