GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-140815: Fix faulthandler for invalid/freed frame by vstinner · Pull Request #140921 · python/cpython Conversation
|
|
I believe this PR minimizes the chances of getting a segfault. However, I have a feeling that it does not eliminate the possibility entirely. Unfortunately, I don't have a better solution. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Perhaps we could add a new function that attempts to attach the thread state with a timeout. If successful, we will have a more reliable solution. WDYT? |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
I don't think that it's worth it. Another faulthandler feature is faulthandler is a debugger, it's not designed to be 100% accurate, but to "just work" in any case (GIL or not, in a signal handler or not, etc.) without taking any lock. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
@sergey-miryanov @picnixz: I pushed a change to address your reviews. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
LGTM, but I have very little expertise here, so I'll not push approve button :-). |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
faulthandler is implemented with multiple heuristics and best effort. It's ok if it does crash, since it should only be triggered when something already goes wrong. Only |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
From point of my little experience, it looks good to me.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Thanks! |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
| if (_PyMem_IsPtrFreed(ptr)) { | ||
| return NULL; | ||
| } | ||
| PyObject *executable = PyStackRef_AsPyObjectBorrow(f->f_executable); | ||
| if (_PyObject_IsFreed(executable)) { |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Does this patch work on release builds? I thought _PyMem_IsPtrFreed and _PyObject_IsFreed only worked if debug allocators are enabled.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Oh. I didn't try on a release build! _PyMem_IsPtrFreed() and _PyObject_IsFreed() work better with debug allocators, you're correct. But they detect freed (reused) memory in some cases in a release build as well. They are just less reliable in release mode.
I pushed more changes to make the heuristics stricter.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
@picnixz @ZeroIntensity: Do these changes look reasonable to you? Even if they are not perfect, they allow dumping the traceback in more cases. I wrote a script to run the reproducer in a loop (see below). I ran the stress test 3 times.
Stress test: import subprocess, sys cmd=[sys.executable, 'reproducer.py'] success = 0 while True: proc = subprocess.run(cmd) if proc.returncode != 1: print() print(f"Exitcode {proc.returncode}") break success += 1 print(success)
import faulthandler import sys faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__) assert False |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
I think this is a good idea anyway, yeah. We might want to document that faulthandler.dump_traceback_later can sometimes crash during interpreter finalization.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| // Similar to _PyFrame_GetCode(), but return NULL if the frame is invalid or | ||
| // freed. Used by dump_frame() in Python/traceback.c. |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
It might be worth adding a comment that this isn't completely reliable.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Good idea. I added a sentence for that in the 3 added functions.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
Merged, thanks for your reviews! |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…40921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <[email protected]>
|
GH-140981 is a backport of this pull request to the 3.14 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
GH-140985 is a backport of this pull request to the 3.13 branch. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)
#140981) gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c) Co-authored-by: Victor Stinner <[email protected]>
…140985) gh-140815: Fix faulthandler for invalid/freed frame (#140921) faulthandler now detects if a frame or a code object is invalid or freed. Add helper functions: * _PyCode_SafeAddr2Line() * _PyFrame_SafeGetCode() * _PyFrame_SafeGetLasti() _PyMem_IsPtrFreed() now detects pointers in [-0xff, 0xff] range as freed. (cherry picked from commit a84181c)
Summary: This diff adds a patch to the `third-party/python/3.12` Meta-internal fork. It backports the trampoline-frame part of the upstream gh-140815 fix to fix a crash in `faulthandler`'s traceback dump. Upstream fixed this in main, 3.14 and 3.13, but not in 3.12, which only takes security fixes. Upstream: - 3.13 backport: python/cpython#140985 - 3.14 backport: python/cpython#140981 (shipped in 3.14.1, already in `third-party/python/3.14`) - main: python/cpython#140921 - issue: python/cpython#140815 The hunks were checked against the 3.14 backport in `third-party/python/3.14/pristine/Python/traceback.c` rather than the 3.13 PR diff. **Problem** A `__del__` that runs while a frame called from C (e.g. by `map()`) is being popped sees two `FRAME_OWNED_BY_CSTACK` shim frames in a row, because a frame is unlinked before it is cleared (GH-99729). `dump_traceback()` skips only one shim and passes the second to `dump_frame()`: - Debug builds fail `assert(frame->owner != FRAME_OWNED_BY_CSTACK)`. - Release builds call `PyUnstable_InterpreterFrame_GetLine()` on the shim. Meta's `_PyFrame_EnsureFrameFullyInitialized()` then reads the shim's `f_funcobj`, which release builds leave uninitialized, and crashes with SIGSEGV. This killed rank 90 of f1134899418. The stuck-job detector's stack-dump signal arrived while a finalizer was in this state, and the dump died right after printing `File "<shim>"`. **Fix** Same as upstream: `dump_frame()` ignores trampoline frames, and the one-shim skip in the `dump_traceback()` loop is removed along with its assert. Adapted to 3.12: - the owner is still `FRAME_OWNED_BY_CSTACK` (3.14 renamed it `FRAME_OWNED_BY_INTERPRETER`); - `dump_frame()` stays `void`, so the early return is `return;` instead of `return 0;`. As upstream, shims now count toward `MAX_FRAME_DEPTH`. The rest of gh-140815 (`_PyFrame_SafeGetCode()`, `_PyFrame_SafeGetLasti()`, `_PyCode_SafeAddr2Line()`, `<freed frame>` / `<invalid frame>`) is not backported. Its frame validation needed a CinderX JIT-frame adaptation in 3.14 (`third-party/python/3.14/patches/21-jit-frame-safe-get-code`) and would need a separate one for 3.12. Upstream added no test for this, so `test_dump_traceback_consecutive_shims` is Meta-local. Reviewed By: itamaro Differential Revision: D121622997 fbshipit-source-id: 7315ccecef7e3432116334db435a0c62a201b332
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.