GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-124375: Avoid calling `_PyMem_ProcessDelayed` on other thread states by colesbury · Pull Request #124459 · python/cpython

Conversation

@colesbury

@colesbury colesbury commented Sep 24, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This fixes a crash when running the PyO3 test suite on the free-threaded build. The qsbr field is initialized after the PyThreadState is added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is always initialized (possibly empty) and protected by the stop the world pause.

  • Issue: Crash running PyO3 tests with --test-threads=1000 #124375

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@colesbury
pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
bc567fc 
…d states

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
@colesbury colesbury added topic-free-threading needs backport to 3.13 only security fixes labels Sep 24, 2024
@colesbury
colesbury requested a review from DinoV September 24, 2024 18:55
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 24, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Sep 24, 2024
Crash running PyO3 tests with --test-threads=1000 #124375
Closed
@colesbury colesbury added the skip news label Sep 24, 2024
@colesbury

colesbury commented Sep 24, 2024

Copy link
Copy Markdown
Contributor Author

I spent a while trying to get a unit test to reproduce the issue without success. I think the period between when HEAD_UNLOCK is called and when qsbr is initialized is pretty short and hard to exploit in a test case:

cpython/Python/pystate.c

Lines 1589 to 1600 in 17b3bc9

HEAD_UNLOCK(runtime);
if (!used_newtstate) {
// Must be called with lock unlocked to avoid re-entrancy deadlock.
PyMem_RawFree(new_tstate);
}
else {
}
#ifdef Py_GIL_DISABLED
// Must be called with lock unlocked to avoid lock ordering deadlocks.
_Py_qsbr_register(tstate, interp, qsbr_idx);
#endif

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury colesbury added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 24, 2024
@bedevere-bot

bedevere-bot commented Sep 24, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @colesbury for commit bc567fc 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 24, 2024
@colesbury

colesbury commented Sep 24, 2024

Copy link
Copy Markdown
Contributor Author

cc @ngoldbaum
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ngoldbaum

ngoldbaum commented Sep 24, 2024

Copy link
Copy Markdown
Contributor

Thanks for this! I'll try to test this with PyO3 tomorrow morning.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ngoldbaum

ngoldbaum commented Sep 25, 2024

Copy link
Copy Markdown
Contributor

Thankfully it seems the 3.14 ABI hasn't yet diverged enough from 3.13 that it's painful to test this with PyO3. It looks like this does fix the segfault - I'm completely unable to trigger it on this PR branch and I can trigger it very quickly with the same test approaches (looping on cargo test --lib -- --test-threads=1000) on 3.13.0rc2 and the current main branch.
colesbury reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
DinoV
DinoV approved these changes Sep 26, 2024
View reviewed changes

@DinoV DinoV left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 26, 2024
blurb-it Bot and others added 2 commits October 15, 2024 15:29
@blurb-it
📜🤖 Added by blurb_it.
ba6a3b3
@colesbury
Merge branch 'main' into pythongh-124375-py-mem-process-delayed
bbaef28
@colesbury
colesbury enabled auto-merge (squash) October 15, 2024 15:31
@colesbury
colesbury merged commit 54c6fcb into python:main Oct 15, 2024
@miss-islington-app

miss-islington-app Bot commented Oct 15, 2024

Copy link
Copy Markdown

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting merge label Oct 15, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 15, 2024
@colesbury @miss-islington
pythongh-124375: Avoid calling _PyMem_ProcessDelayed on other threa…
93e51c9 
…d states (pythonGH-124459)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <[email protected]>
@bedevere-app

bedevere-app Bot commented Oct 15, 2024

Copy link
Copy Markdown

GH-125540 is a backport of this pull request to the 3.13 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Oct 15, 2024
colesbury added a commit that referenced this pull request Oct 15, 2024
@miss-islington @colesbury
[3.13] gh-124375: Avoid calling _PyMem_ProcessDelayed on other thre…
4d83f6d 
…ad states (GH-124459) (#125540)

This fixes a crash when running the PyO3 test suite on the free-threaded
build. The `qsbr` field is initialized after the `PyThreadState` is
added to the interpreter's linked list -- it might still be NULL.

Instead, we "steal" the queue of to-be-freed memory blocks. This is
always initialized (possibly empty) and protected by the stop the world
pause.
(cherry picked from commit 54c6fcb)

Co-authored-by: Sam Gross <[email protected]>
@colesbury
colesbury deleted the gh-124375-py-mem-process-delayed branch December 4, 2024 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DinoV DinoV DinoV approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@colesbury @bedevere-bot @ngoldbaum @DinoV
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.