GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-129185: Fix PyTraceMalloc_Untrack() at Python exit by vstinner · Pull Request #129191 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Jan 22, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack() during late Python finalization.

  • Issue: Segmentation fault during interpreter finalization in PyTraceMalloc_Untrack #129185

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
pythongh-129185: Fix PyTraceMalloc_Untrack() at Python exit
a1ea2fe
Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack()
during late Python finalization.
@vstinner vstinner added the skip news label Jan 22, 2025
@bedevere-app bedevere-app Bot added the awaiting core review label Jan 22, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Jan 22, 2025
Segmentation fault during interpreter finalization in PyTraceMalloc_Untrack #129185
Closed
@vstinner vstinner mentioned this pull request Jan 22, 2025
gh-129185: Fix crash during interpreter finalization in PyTraceMalloc_Untrack #129188
Closed
@vstinner vstinner added needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes labels Jan 22, 2025
@vstinner

vstinner commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

cc @ZeroIntensity

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity reviewed Jan 22, 2025
View reviewed changes
Comment thread Python/tracemalloc.c
size_t size)
{
// gh-129185: Pre-check to support calls after _PyTraceMalloc_Fini()
if (!tracemalloc_config.tracing) {

@ZeroIntensity ZeroIntensity Jan 22, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to hold the lock or GIL, right?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 22, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of this "pre-check" is to avoid the GIL and TABLES_LOCK().

The flag is tested again below with TABLES_LOCK().

PyTraceMalloc_Untrack() doesn't lock the GIL, only TABLES_LOCK(), when tracing.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 22, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change restores the check that we had before recent refactoring. For example, Python 3.14 alpha3 uses:

int
PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr,
                    size_t size)
{
    int res;
    PyGILState_STATE gil_state;

    if (!tracemalloc_config.tracing) {
        /* tracemalloc is not tracing: do nothing */
        return -2;
    }

    gil_state = PyGILState_Ensure();

    TABLES_LOCK();
    res = tracemalloc_add_trace(domain, ptr, size);
    TABLES_UNLOCK();

    PyGILState_Release(gil_state);
    return res;
}

The check is done without holding the GIL.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Jan 23, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then we lose the thread safety: if one thread that holds the GIL were to write to tracing, then a thread that calls one of these without the GIL would race.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Jan 23, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the easiest solution would be to grab the tables lock for the read, and then unlock it before calling PyGILState_Ensure to prevent lock-ordering deadlocks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 23, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The race is right here. Thread A might check tracing at the exact same time as thread B, which is a data race.

What is the problem of "continue execution" of thread A, since thread A checks again tracing with the tables lock?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Jan 23, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it might not reach the second tracing read at all--if the first one races, then we're probably going to get a crash.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 23, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, you're talking about thread B calling _PyTraceMalloc_Fini(), not tracemalloc.stop(). In this case, thread A can call TABLES_LOCK() after thread B deleted the lock, and yes, we can get a crash.

Sadly, PyTraceMalloc_Track() and PyTraceMalloc_Untrack() API doesn't require the caller to hold the GIL.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ZeroIntensity ZeroIntensity Jan 23, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That can happen too, but I meant that we'll get a crash from the data race on tracing, not a use-after-free on the lock. I think we just hold the GIL (via PyGILState_Ensure) and call without the tables lock held, that should be thread safe enough for 3.12.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Jan 23, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified the Untrack() function to get the GIL. It should address your last concern.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
vstinner added 3 commits January 23, 2025 02:41
@vstinner
Test also PyTraceMalloc_Untrack() without the GIL
4172b91
@vstinner
Call _PyTraceMalloc_Fini() after finalize_interp_clear()
f3ec095
@vstinner
PyTraceMalloc_Track() checks tracing with the GIL
d510567
@vstinner
vstinner requested a review from ericsnowcurrently as a code owner January 23, 2025 01:44
@vstinner

vstinner commented Jan 23, 2025

Copy link
Copy Markdown
Member Author

@ZeroIntensity: Please review the updated PR, it should reduce the risk of race.

#129185 is not about threads running in parallel. It's just a single thread and PyTraceMalloc_Track/Untrack() called by Python objects destroyed after _PyTraceMalloc_Fini().

I propose to merge this incomplete fix and backport it, and then consider switching to PyMutex in the main branch to avoid the last race.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added 2 commits January 23, 2025 02:57
@vstinner
Test also Untrack in test_tracemalloc_track_race()
31bfe5b
@vstinner
Get the GIL in Untrack
555c70d
@ZeroIntensity

ZeroIntensity commented Jan 23, 2025

Copy link
Copy Markdown
Member

#129185 is not about threads running in parallel. It's just a single thread and PyTraceMalloc_Track/Untrack() called by Python objects destroyed after _PyTraceMalloc_Fini().

Yeah, I'm talking about the other issue that you recently fixed. The previous approach would make it a problem again.

I propose to merge this incomplete fix and backport it, and then consider switching to PyMutex in the main branch to avoid the last race.

I agree.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity approved these changes Jan 23, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
vstinner merged commit 46c7e13 into python:main Jan 23, 2025
@vstinner
vstinner deleted the tracemalloc branch January 23, 2025 11:07
@bedevere-app bedevere-app Bot removed the awaiting core review label Jan 23, 2025
@miss-islington-app

miss-islington-app Bot commented Jan 23, 2025

Copy link
Copy Markdown

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app

miss-islington-app Bot commented Jan 23, 2025

Copy link
Copy Markdown

Sorry, @vstinner, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 46c7e13c055c218e18b0424efc60965e6a5fe6ea 3.13
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington-app miss-islington-app Bot assigned vstinner Jan 23, 2025
@miss-islington-app

miss-islington-app Bot commented Jan 23, 2025

Copy link
Copy Markdown

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 46c7e13c055c218e18b0424efc60965e6a5fe6ea 3.12
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added a commit to vstinner/cpython that referenced this pull request Jan 23, 2025
@vstinner
pythongh-129185: Fix PyTraceMalloc_Untrack() at Python exit (python#1…
62eebd3
…29191)

Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack()
during late Python finalization.

* Call _PyTraceMalloc_Fini() later in Python finalization.
* Test also PyTraceMalloc_Untrack() without the GIL
* PyTraceMalloc_Untrack() now gets the GIL.
* Test also PyTraceMalloc_Untrack() in test_tracemalloc_track_race().

(cherry picked from commit 46c7e13)
@bedevere-app

bedevere-app Bot commented Jan 23, 2025

Copy link
Copy Markdown

GH-129217 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Jan 23, 2025
@vstinner vstinner removed the needs backport to 3.12 only security fixes label Jan 23, 2025
vstinner added a commit that referenced this pull request Jan 23, 2025
@vstinner
[3.13] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (
e3b3e01
#129217)

gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191)

Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack()
during late Python finalization.

* Call _PyTraceMalloc_Fini() later in Python finalization.
* Test also PyTraceMalloc_Untrack() without the GIL
* PyTraceMalloc_Untrack() now gets the GIL.
* Test also PyTraceMalloc_Untrack() in test_tracemalloc_track_race().

(cherry picked from commit 46c7e13)
@vstinner vstinner mentioned this pull request Jan 23, 2025
[3.12] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (#129217) #129221
Merged
vstinner added a commit that referenced this pull request Jan 23, 2025
@vstinner
[3.12] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (
bb7c54d
#129217) (#129221)

[3.13] gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191) (#129217)

gh-129185: Fix PyTraceMalloc_Untrack() at Python exit (#129191)

Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack()
during late Python finalization.

* Call _PyTraceMalloc_Fini() later in Python finalization.
* Test also PyTraceMalloc_Untrack() without the GIL
* PyTraceMalloc_Untrack() now gets the GIL.
* Test also PyTraceMalloc_Untrack() in test_tracemalloc_track_race().

(cherry picked from commit 46c7e13)
(cherry picked from commit e3b3e01)
@ZeroIntensity ZeroIntensity mentioned this pull request Jan 24, 2025
gh-129185: Simplify PyTraceMalloc_Track() #129256
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@vstinner vstinner

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@vstinner @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.