GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters by ericsnowcurrently · Pull Request #107567 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Aug 2, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.

  • Issue: test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain #107080

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ericsnowcurrently added 4 commits August 2, 2023 10:39
@ericsnowcurrently
Move refchain to _PyRuntimeState.
11d48cd
@ericsnowcurrently
Move refchain to PyInterpreterState.
238e00f
@ericsnowcurrently
Add a NEWS entry.
33105c0
@ericsnowcurrently
Clean up _Py_GetObjects().
ac78d7b
@ericsnowcurrently ericsnowcurrently added the needs backport to 3.12 only security fixes label Aug 2, 2023
@bedevere-bot bedevere-bot mentioned this pull request Aug 2, 2023
test__xxsubinterpreters: test_already_running() crash randomly on Python built with TraceRefs: invalid object chain #107080
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Aug 2, 2023
@ericsnowcurrently
ericsnowcurrently requested a review from vstinner August 2, 2023 21:16
@ericsnowcurrently ericsnowcurrently mentioned this pull request Aug 3, 2023
gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL #107586
Merged
Yhg1s
Yhg1s reviewed Aug 3, 2023
View reviewed changes
Comment thread Objects/object.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/pylifecycle.c Outdated
}

PyObject refchain = { 0 };
_Py_StealRefchain(tstate->interp, &refchain);

@Yhg1s Yhg1s Aug 3, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ericsnowcurrently ericsnowcurrently Aug 3, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interpreter will be deallocated before we later call _Py_PrintReferenceAddresses() to walk the linked list.

That said, I think there's a related problem that I still need to address. Namely, every remaining object in the linked list will be (effectively) freed when the interpreter's allocator is finalized, so the _ob_next and _ob_prev accesses might segfault. It likely isn't a big problem during runtime finalization (except perhaps for embedders) but I'll fix that regardless.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ericsnowcurrently ericsnowcurrently Aug 3, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm pretty sure that's not actually a problem. We only call _Py_PrintReferenceAddresses() in Py_FinalizeEx() and only for the main interpreter, which is statically allocated. Objects in the linked list might still cause a segfault but only if they would have before this change, so I'm not going to worry about that here. I will leave a note in the code though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ericsnowcurrently
Drop _Py_StealRefchain().
e21c558
@ericsnowcurrently ericsnowcurrently mentioned this pull request Aug 3, 2023
gh-105699: Re-enable the Multiple-Interpreters Stress Tests #107572
Merged
@ericsnowcurrently
Drop _Py_StealRefchain().
7a8e549
Yhg1s
Yhg1s approved these changes Aug 3, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Aug 3, 2023
@ericsnowcurrently
ericsnowcurrently enabled auto-merge (squash) August 3, 2023 19:43
@ericsnowcurrently
ericsnowcurrently merged commit 58ef741 into python:main Aug 3, 2023
@miss-islington

miss-islington commented Aug 3, 2023

Copy link
Copy Markdown
Contributor

Thanks @ericsnowcurrently for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting merge label Aug 3, 2023
@bedevere-bot

bedevere-bot commented Aug 3, 2023

Copy link
Copy Markdown

GH-107599 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label Aug 3, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 3, 2023
@ericsnowcurrently @miss-islington
pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpre…
b55de7a
…ters (pythongh-107567)

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
(cherry picked from commit 58ef741)

Co-authored-by: Eric Snow <[email protected]>
@ericsnowcurrently
ericsnowcurrently deleted the fix-tracerefs branch August 3, 2023 20:02
Yhg1s pushed a commit that referenced this pull request Aug 3, 2023
@miss-islington @ericsnowcurrently
[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…
58af229
…eters (gh-107567) (#107599)

gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567)

The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
(cherry picked from commit 58ef741)

Co-authored-by: Eric Snow <[email protected]>
@Yhg1s Yhg1s mentioned this pull request Aug 4, 2023
TraceRefs build triggers assertion errors #107630
Closed
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this pull request Aug 4, 2023
@ericsnowcurrently
Revert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Isola…
b380392
…ted Subinterpreters (pythongh-107567) (python#107599)"

This reverts commit 58af229.
Yhg1s pushed a commit that referenced this pull request Aug 5, 2023
@ericsnowcurrently
[3.12] gh-107630: Revert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes…
6e4eec7
… Under Isolated Subinterpreters (gh-107567) (#107599)" (#107648)

Revert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567) (#107599)"

This reverts commit 58af229.
@ericsnowcurrently ericsnowcurrently mentioned this pull request Aug 7, 2023
gh-107630: Initialize Each Interpreter's refchain Properly #107733
Merged
ericsnowcurrently added a commit to ericsnowcurrently/cpython that referenced this pull request Aug 7, 2023
@ericsnowcurrently
Unrevert "[3.12] pythongh-107080: Fix Py_TRACE_REFS Crashes Under Iso…
6f8b84f
…lated Subinterpreters (pythongh-107567) (python#107599)".

This reverts commit 6e4eec7 (pythongh-107648).
ericsnowcurrently added a commit that referenced this pull request Aug 7, 2023
@ericsnowcurrently
gh-107630: Initialize Each Interpreter's refchain Properly (gh-107733)
430632d
This finishes fixing the crashes in Py_TRACE_REFS builds.  We missed this part in gh-107567.
@ericsnowcurrently ericsnowcurrently mentioned this pull request Aug 7, 2023
gh-107630: Fix Remaining Subinterpreters Crashes on Py_TRACE_REFS Builds #107750
Merged
ericsnowcurrently added a commit that referenced this pull request Aug 7, 2023
@ericsnowcurrently
gh-107630: Fix Remaining Subinterpreters Crashes on Py_TRACE_REFS Bui…
707018c
…lds (gh-107750)

This is a follow-up to gh-107567 and gh-107733.

We skip test_basic_multiple_interpreters_deleted_no_reset on tracerefs builds.  The test breaks interpreter isolation a little, which doesn't work well with Py_TRACE_REFS builds, so I feel fine about skipping the test.
Yhg1s pushed a commit that referenced this pull request Aug 16, 2023
@ericsnowcurrently
[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpr…
aa9707d
…eters (#107751)

* Unrevert "[3.12] gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567) (#107599)".

This reverts commit 6e4eec7 (gh-107648).

* Initialize each interpreter's refchain properly.

* Skip test_basic_multiple_interpreters_deleted_no_reset on tracerefs builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yhg1s Yhg1s Yhg1s approved these changes
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@ericsnowcurrently @miss-islington @bedevere-bot @Yhg1s
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.