GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-128679: Redesign tracemalloc locking by vstinner · Pull Request #128888 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Jan 15, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Use TABLES_LOCK() to protect 'tracemalloc_config.tracing'.
  • Hold TABLES_LOCK() longer while accessing tables.
  • tracemalloc_realloc_gil() and tracemalloc_raw_realloc() no longer remove the trace on reentrant call.
  • _PyTraceMalloc_Stop() unregisters _PyTraceMalloc_TraceRef().
  • _PyTraceMalloc_GetTraces() sets the reentrant flag.
  • tracemalloc_clear_traces_unlocked() sets the reentrant flag.
  • Issue: Race condition in tracemalloc causes segfaults #128679

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@vstinner
pythongh-128679: Redesign tracemalloc locking
a415ecb 
* Use TABLES_LOCK() to protect 'tracemalloc_config.tracing'.
* Hold TABLES_LOCK() longer while accessing tables.
* tracemalloc_realloc_gil() and tracemalloc_raw_realloc() no longer
  remove the trace on reentrant call.
* _PyTraceMalloc_Stop() unregisters _PyTraceMalloc_TraceRef().
* _PyTraceMalloc_GetTraces() sets the reentrant flag.
* tracemalloc_clear_traces_unlocked() sets the reentrant flag.
@vstinner vstinner added the skip news label Jan 15, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Jan 15, 2025
Race condition in tracemalloc causes segfaults #128679
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Jan 15, 2025
@vstinner vstinner mentioned this pull request Jan 15, 2025
gh-128679: Fix tracemalloc.stop() race condition #128710
Closed
@vstinner

vstinner commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

_PyTraceMalloc_Stop() unregisters _PyTraceMalloc_TraceRef().

This change cannot be backported to Python 3.12 which doesn't have the PyRefTracer_SetTracer() function.

Anyway, I don't plan to backport this major redesign to Python 3.13 neither.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
Call _PyTraceMalloc_Init() earlier in Python init
867f20b
@vstinner
vstinner requested a review from ericsnowcurrently as a code owner January 15, 2025 17:11
vstinner added 2 commits January 15, 2025 19:49
@vstinner
Fix tracemalloc_free()
d7ffacf 
Don't remove the trace if it's a reentrant call.
@vstinner
Simplify alloc/realloc/free code
fd2c44c
ZeroIntensity
ZeroIntensity reviewed Jan 15, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a much better solution. A few notes:

  • I think it's worth trying to migrate away from PyThread, because it's a lot more difficult to work with than the modern APIs. We can go with PyMutex and _Py_thread_local instead.
  • We need to be careful with holding locks while acquiring the GIL, because that can lead to lock-ordering deadlocks.
  • I'll need to look into what kind of subinterpreter problems might come up because of PyGILState_Ensure, as it doesn't work very well. What would happen if e.g. a thread calls PyTraceMalloc_Track, which sets the GILstate for the main interpreter, then enters a different interpreter?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@vstinner
Fix tracemalloc_alloc/realloc() reentrant call
17ebde5
@vstinner
vstinner enabled auto-merge (squash) January 15, 2025 20:16
@vstinner
vstinner merged commit 36c5e3b into python:main Jan 15, 2025
@vstinner
vstinner deleted the tracemalloc_lock branch January 15, 2025 20:22
@bedevere-app bedevere-app Bot removed the awaiting core review label Jan 15, 2025
@vstinner

vstinner commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

I think it's worth trying to migrate away from PyThread, because it's a lot more difficult to work with than the modern APIs. We can go with PyMutex and _Py_thread_local instead.

Why not. It can be done afterwards.

We need to be careful with holding locks while acquiring the GIL, because that can lead to lock-ordering deadlocks.

Right. The order is: first get the GIL, and then call TABLES_LOCK().

I'll need to look into what kind of subinterpreter problems might come up because of PyGILState_Ensure, as #123728. What would happen if e.g. a thread calls PyTraceMalloc_Track, which sets the GILstate for the main interpreter, then enters a different interpreter?

This PR doesn't change that. But right, there should be corner cases when sub-interpreters are involved.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner vstinner mentioned this pull request Jan 15, 2025
gh-128679: Fix tracemalloc.stop() race conditions #128893
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ZeroIntensity ZeroIntensity ZeroIntensity left review comments
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@vstinner @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.