GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-123022: Fix crash with `Py_Initialize` in background thread by colesbury · Pull Request #123052 · python/cpython

Conversation

@colesbury

@colesbury colesbury commented Aug 15, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Check that the current default heap is initialized in _mi_os_get_aligned_hint and mi_os_claim_huge_pages.

The mimalloc function _mi_os_get_aligned_hint assumes that there is an initialized default heap. This is true for our main thread, but not for background threads. The problematic code path is usually called during initialization (i.e., Py_Initialize), but it may also be called if the program allocates large amounts of memory in total.

The crash only affected the free-threaded build.

  • Issue: Crash in Py_Initialize in non-main thread in free-threading build #123022

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@colesbury
pythongh-123022: Fix crash with Py_Initialize in background thread
8d554b4 
Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
@colesbury colesbury added topic-free-threading needs backport to 3.13 only security fixes labels Aug 15, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Aug 15, 2024
Crash in Py_Initialize in non-main thread in free-threading build #123022
Closed
@colesbury colesbury mentioned this pull request Aug 15, 2024
Mimalloc differences from upstream #113141
Open
@colesbury

colesbury commented Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

We may want to consider setting the default heap in the future, but that would be a more involved change because CPython's thread destruction process doesn't line up perfectly with mimalloc's model. In particular, PyThreadState instances may be destroyed by different threads, while mimalloc assumes that the thread's default heap is destroyed when the OS thread exits.

The approach in this PR -- guarding the call to _mi_heap_random -- seemed like it would have the least risk of complications.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury
colesbury marked this pull request as ready for review August 15, 2024 20:19
@bedevere-app bedevere-app Bot added the awaiting core review label Aug 15, 2024
@colesbury
colesbury requested review from Yhg1s and vstinner August 15, 2024 20:21
@colesbury

colesbury commented Aug 15, 2024

Copy link
Copy Markdown
Contributor Author

@Yhg1s - I think it's worth considering this for 3.13rc2. Let me know what you think.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury colesbury mentioned this pull request Aug 15, 2024
Tracking issue for no-gil/freethreaded work PyO3/pyo3#4265
Closed
14 tasks done
@Yhg1s

Yhg1s commented Aug 17, 2024

Copy link
Copy Markdown
Member

@Yhg1s - I think it's worth considering this for 3.13rc2. Let me know what you think.

Yeah, this should go into rc2.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Yhg1s
Yhg1s approved these changes Aug 17, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Aug 17, 2024
@colesbury
colesbury merged commit d061ffe into python:main Aug 17, 2024
@colesbury
colesbury deleted the gh-123022-mi-prim_get_default branch August 17, 2024 20:04
@bedevere-app bedevere-app Bot removed the awaiting merge label Aug 17, 2024
@miss-islington-app

miss-islington-app Bot commented Aug 17, 2024

Copy link
Copy Markdown

Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 17, 2024
@colesbury @miss-islington
pythongh-123022: Fix crash with Py_Initialize in background thread (p…
ca630c3 
…ythonGH-123052)

Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
(cherry picked from commit d061ffe)

Co-authored-by: Sam Gross <[email protected]>
@bedevere-app

bedevere-app Bot commented Aug 17, 2024

Copy link
Copy Markdown

GH-123114 is a backport of this pull request to the 3.13 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Aug 17, 2024
colesbury added a commit that referenced this pull request Aug 17, 2024
@miss-islington @colesbury
[3.13] gh-123022: Fix crash with Py_Initialize in background thread (…
ff58c3c 
…GH-123052) (#123114)

Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
(cherry picked from commit d061ffe)

Co-authored-by: Sam Gross <[email protected]>
jeremyhylton pushed a commit to jeremyhylton/cpython that referenced this pull request Aug 19, 2024
@colesbury @jeremyhylton
pythongh-123022: Fix crash with Py_Initialize in background thread (p…
55f90c8 
…ython#123052)

Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
blhsing pushed a commit to blhsing/cpython that referenced this pull request Aug 22, 2024
@colesbury @blhsing
pythongh-123022: Fix crash with Py_Initialize in background thread (p…
84e517f 
…ython#123052)

Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yhg1s Yhg1s Yhg1s approved these changes
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@colesbury @Yhg1s
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.