GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-140815: Fix faulthandler for invalid/freed frame by vstinner · Pull Request #140921 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Nov 2, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

faulthandler now detects if a frame or a code object is invalid or freed.

Add helper functions:

  • _PyCode_SafeAddr2Line()
  • _PyFrame_SafeGetCode()
  • _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.

  • Issue: SEGV in faulthandler.dump_traceback_later #140815

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
pythongh-140815: Fix faulthandler for invalid/freed frame
1cab1e5
faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetBytecode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now also considers the pointer 0x1 as freed.
@vstinner
vstinner requested review from ericsnowcurrently and markshannon as code owners November 2, 2025 18:15
@bedevere-app bedevere-app Bot added the awaiting core review label Nov 2, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Nov 2, 2025
SEGV in faulthandler.dump_traceback_later #140815
Closed
@vstinner
Add NEWS entry
4059837
@vstinner vstinner mentioned this pull request Nov 2, 2025
GH-140815: Acquire GIL or stop the world when calling _Py_DumpTracebackThreads from faulthandler_thread #140895
Closed
sergey-miryanov
sergey-miryanov reviewed Nov 2, 2025
View reviewed changes
Comment thread Include/internal/pycore_interpframe.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

sergey-miryanov
sergey-miryanov reviewed Nov 2, 2025
View reviewed changes
Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sergey-miryanov

sergey-miryanov commented Nov 2, 2025

Copy link
Copy Markdown
Contributor

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sergey-miryanov

sergey-miryanov commented Nov 2, 2025

Copy link
Copy Markdown
Contributor

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Nov 2, 2025
View reviewed changes
Comment thread Include/internal/pycore_interpframe.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Include/internal/pycore_interpframe.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Include/internal/pycore_interpframe.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 2, 2025

Copy link
Copy Markdown
Member Author

Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT?

I don't think that it's worth it.

Another faulthandler feature is faulthandler.enable() and faulthandler.register(): register signal handlers. A Unix signal can occur anytime, so we cannot acquire the GIL or any other lock.

faulthandler is a debugger, it's not designed to be 100% accurate, but to "just work" in any case (GIL or not, in a signal handler or not, etc.) without taking any lock.

sergey-miryanov reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Address reviews
9b26187
@vstinner

vstinner commented Nov 2, 2025

Copy link
Copy Markdown
Member Author

@sergey-miryanov @picnixz: I pushed a change to address your reviews.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

efimov-mikhail
efimov-mikhail reviewed Nov 2, 2025
View reviewed changes
Comment thread Include/internal/pycore_code.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Update Include/internal/pycore_code.h
7d9a4b8
@efimov-mikhail

efimov-mikhail commented Nov 2, 2025

Copy link
Copy Markdown
Member

LGTM, but I have very little expertise here, so I'll not push approve button :-).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 2, 2025

Copy link
Copy Markdown
Member Author

I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution.

faulthandler is implemented with multiple heuristics and best effort. It's ok if it does crash, since it should only be triggered when something already goes wrong.

Only faulthandler.dump_traceback() should be safe since it's called with an attached thread safe.

sergey-miryanov reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sergey-miryanov
sergey-miryanov approved these changes Nov 2, 2025
View reviewed changes

@sergey-miryanov sergey-miryanov left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From point of my little experience, it looks good to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sergey-miryanov

sergey-miryanov commented Nov 2, 2025

Copy link
Copy Markdown
Contributor

Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Nov 2, 2025
View reviewed changes
Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity reviewed Nov 2, 2025
View reviewed changes
Comment thread Include/internal/pycore_interpframe.h
Comment on lines +37 to +41
if (_PyMem_IsPtrFreed(ptr)) {
return NULL;
}
PyObject *executable = PyStackRef_AsPyObjectBorrow(f->f_executable);
if (_PyObject_IsFreed(executable)) {

@ZeroIntensity ZeroIntensity Nov 2, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this patch work on release builds? I thought _PyMem_IsPtrFreed and _PyObject_IsFreed only worked if debug allocators are enabled.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Nov 3, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. I didn't try on a release build! _PyMem_IsPtrFreed() and _PyObject_IsFreed() work better with debug allocators, you're correct. But they detect freed (reused) memory in some cases in a release build as well. They are just less reliable in release mode.

I pushed more changes to make the heuristics stricter.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner added 3 commits November 3, 2025 12:44
@vstinner
Enhance _PyMem_IsPtrFreed()
c208539
@vstinner
Stricter _PyFrame_SafeGetCode()
c273f99
@vstinner
coding style
3764548
@vstinner

vstinner commented Nov 3, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@picnixz @ZeroIntensity: Do these changes look reasonable to you? Even if they are not perfect, they allow dumping the traceback in more cases.

I wrote a script to run the reproducer in a loop (see below). I ran the stress test 3 times.

  • Debug build:
    • Without these changes: crash after 1 run, crash after 6 runs, crash after 1 run
    • With these changes: crash after 3970 runs, <stopped after 10,000 success without crash>, <stopped after 10,000 success without crash>.
  • Release build:
    • Without these changes: crash after 6 runs, crash after 59 runs, crash after 36 runs
    • With these changes: crash after 550 runs, crash after 3750 runs, crash after 7845 runs.

Stress test:

import subprocess, sys
cmd=[sys.executable, 'reproducer.py']
success = 0
while True:
    proc = subprocess.run(cmd)
    if proc.returncode != 1:
        print()
        print(f"Exitcode {proc.returncode}")
        break
    success += 1
    print(success)

reproducer.py:

import faulthandler
import sys
faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
assert False
efimov-mikhail reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity approved these changes Nov 3, 2025
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good idea anyway, yeah. We might want to document that faulthandler.dump_traceback_later can sometimes crash during interpreter finalization.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Include/internal/pycore_interpframe.h Outdated
Comment on lines +27 to +28
// Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or
// freed. Used by dump_frame() in Python/traceback.c.

@ZeroIntensity ZeroIntensity Nov 3, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth adding a comment that this isn't completely reliable.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Nov 3, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I added a sentence for that in the 3 added functions.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Nov 3, 2025
@vstinner
Document that safe functions are not 100% reliable
6239435
@vstinner
vstinner merged commit a84181c into python:main Nov 4, 2025
4 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner deleted the faulthandler_freed branch November 4, 2025 10:48
@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 4, 2025
@vstinner

vstinner commented Nov 4, 2025

Copy link
Copy Markdown
Member Author

Merged, thanks for your reviews!

efimov-mikhail reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner added the needs backport to 3.14 bugs and security fixes label Nov 4, 2025
@miss-islington-app

miss-islington-app Bot commented Nov 4, 2025

Copy link
Copy Markdown

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 4, 2025
@vstinner @miss-islington
pythongh-140815: Fix faulthandler for invalid/freed frame (pythonGH-1…
ed39713
…40921)

faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
(cherry picked from commit a84181c)

Co-authored-by: Victor Stinner <[email protected]>
@bedevere-app

bedevere-app Bot commented Nov 4, 2025

Copy link
Copy Markdown

GH-140981 is a backport of this pull request to the 3.14 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Nov 4, 2025
@bedevere-app

bedevere-app Bot commented Nov 4, 2025

Copy link
Copy Markdown

GH-140985 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added a commit to vstinner/cpython that referenced this pull request Nov 4, 2025
@vstinner
pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921
1aee44b
)

faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.

(cherry picked from commit a84181c)
vstinner added a commit that referenced this pull request Nov 4, 2025
@miss-islington @vstinner
[3.14] gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) (
331b4b8
#140981)

gh-140815: Fix faulthandler for invalid/freed frame (GH-140921)

faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
(cherry picked from commit a84181c)

Co-authored-by: Victor Stinner <[email protected]>
@sergey-miryanov sergey-miryanov mentioned this pull request Nov 5, 2025
Segfault in faulthandler signal handler with threads #116008
Closed
vstinner added a commit that referenced this pull request Nov 5, 2025
@vstinner
[3.13] gh-140815: Fix faulthandler for invalid/freed frame (#140921) (#…
43882c7
…140985)

gh-140815: Fix faulthandler for invalid/freed frame (#140921)

faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.

(cherry picked from commit a84181c)
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@vstinner @StanFromIreland
pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921
592b099
)

faulthandler now detects if a frame or a code object is invalid or
freed.

Add helper functions:

* _PyCode_SafeAddr2Line()
* _PyFrame_SafeGetCode()
* _PyFrame_SafeGetLasti()

_PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range
as freed.
meta-codesync Bot pushed a commit to facebookincubator/MetaPython that referenced this pull request Sep 24, 2026
@stephenzhang-meta @meta-codesync
Backport gh-140815 faulthandler trampoline-frame fix
247b988
Summary:
This diff adds a patch to the `third-party/python/3.12` Meta-internal fork. It backports the trampoline-frame part of the upstream gh-140815 fix to fix a crash in `faulthandler`'s traceback dump. Upstream fixed this in main, 3.14 and 3.13, but not in 3.12, which only takes security fixes.

Upstream:
- 3.13 backport: python/cpython#140985
- 3.14 backport: python/cpython#140981 (shipped in 3.14.1, already in `third-party/python/3.14`)
- main: python/cpython#140921
- issue: python/cpython#140815

The hunks were checked against the 3.14 backport in `third-party/python/3.14/pristine/Python/traceback.c` rather than the 3.13 PR diff.

**Problem**
A `__del__` that runs while a frame called from C (e.g. by `map()`) is being popped sees two `FRAME_OWNED_BY_CSTACK` shim frames in a row, because a frame is unlinked before it is cleared (GH-99729). `dump_traceback()` skips only one shim and passes the second to `dump_frame()`:
- Debug builds fail `assert(frame->owner != FRAME_OWNED_BY_CSTACK)`.
- Release builds call `PyUnstable_InterpreterFrame_GetLine()` on the shim. Meta's `_PyFrame_EnsureFrameFullyInitialized()` then reads the shim's `f_funcobj`, which release builds leave uninitialized, and crashes with SIGSEGV.

This killed rank 90 of f1134899418. The stuck-job detector's stack-dump signal arrived while a finalizer was in this state, and the dump died right after printing `File "<shim>"`.

**Fix**
Same as upstream: `dump_frame()` ignores trampoline frames, and the one-shim skip in the `dump_traceback()` loop is removed along with its assert. Adapted to 3.12:
- the owner is still `FRAME_OWNED_BY_CSTACK` (3.14 renamed it `FRAME_OWNED_BY_INTERPRETER`);
- `dump_frame()` stays `void`, so the early return is `return;` instead of `return 0;`.

As upstream, shims now count toward `MAX_FRAME_DEPTH`.

The rest of gh-140815 (`_PyFrame_SafeGetCode()`, `_PyFrame_SafeGetLasti()`, `_PyCode_SafeAddr2Line()`, `<freed frame>` / `<invalid frame>`) is not backported. Its frame validation needed a CinderX JIT-frame adaptation in 3.14 (`third-party/python/3.14/patches/21-jit-frame-safe-get-code`) and would need a separate one for 3.12.

Upstream added no test for this, so `test_dump_traceback_consecutive_shims` is Meta-local.

Reviewed By: itamaro

Differential Revision: D121622997

fbshipit-source-id: 7315ccecef7e3432116334db435a0c62a201b332
@Kosinkadink Kosinkadink mentioned this pull request Oct 31, 2026
Diagnose Windows generation repair CLI timeouts Kosinkadink/Dinkster#762
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efimov-mikhail efimov-mikhail efimov-mikhail left review comments
@picnixz picnixz picnixz left review comments
@sergey-miryanov sergey-miryanov sergey-miryanov approved these changes
@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@vstinner @sergey-miryanov @efimov-mikhail @picnixz @ZeroIntensity
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.