GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Frame teardown can create frame objects · Issue #99729 · python/cpython

Frame teardown can create frame objects #99729

New issue

Copy link

New issue

Copy link

Closed

Frame teardown can create frame objects#99729

Copy link

Assignees

Labels

3.11only security fixesonly security fixes3.12only security fixesonly security fixestype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

Description

graingert

opened

on Nov 23, 2022

Issue body actions

Crash report

using https://github.com/graingert/segfault-repro running pytest yields a segfault in about 1 in 3 runs

================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/graingert/projects/segfault-repro, configfile: pyproject.toml
collected 1 item                                                                                                                                                                                                                                                                         

test_ssltransport.py Fatal Python error: Segmentation fault

Current thread 0x00007f874b2df000 (most recent call first):
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 43 in _hook
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/python.py", line 1789 in runtest
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 339 in from_call
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/main.py", line 324 in _main
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/home/graingert/.virtualenvs/segfault-repro/lib/python3.11/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/home/graingert/.virtualenvs/segfault-repro/bin/pytest", line 8 in <module>
[1]    50315 segmentation fault (core dumped)  pytest

Error messages

Enter any relevant error message caused by the crash, including a core dump if there is one.

Your environment

CPython versions tested on: Python 3.11.0 (main, Oct 24 2022, 19:56:13) [GCC 11.2.0] on linux
Operating system and architecture: 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Linked PRs

Activity

graingert
added
type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
on Nov 23, 2022
graingert commented on Nov 23, 2022
graingert
on Nov 23, 2022
ContributorAuthor
More actions
see also bloomberg/memray#260 cc @pablogsal
graingert
assigned
pablogsal
on Nov 23, 2022

pablogsal commented on Nov 23, 2022

pablogsal

on Nov 23, 2022

Member

More actions

This looks like another frame lifetime issue. Here is the stack:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7cc5740 (LWP 223005)]
0x00005555556cbc8b in PyFrame_GetCode (frame=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>) at Objects/frameobject.c:1303
1303    Objects/frameobject.c: Directory not empty.
(gdb) up
#1  frame_getcode (f=<error reading variable: Cannot access memory at address 0x7ffff5e4c210>, closure=<optimized out>)
    at Objects/frameobject.c:97
97      in Objects/frameobject.c
(gdb)
#2  0x0000555555702309 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'>: There is no member named f_frame.
obj=, name='f_code', dict=0x0, suppress=0) at ./Include/object.h:133
133     ./Include/object.h: Directory not empty.
(gdb)
#3  0x0000555555701836 in PyObject_GetAttr (Python Exception <class 'gdb.error'>: There is no member named f_frame.
v=v@entry=, name='f_code') at Objects/object.c:916
916     Objects/object.c: Directory not empty.
(gdb)
#4  0x00005555556440a1 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3466
3466    Python/ceval.c: Directory not empty.
(gdb)
#5  0x00005555556c1bd5 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff4840330, tstate=0x555555adfa18 <_PyRuntime+166328>)
    at ./Include/internal/pycore_ceval.h:73
73      ./Include/internal/pycore_ceval.h: Directory not empty.
(gdb)
#6  gen_send_ex2 (closing=0, exc=0, presult=<synthetic pointer>, arg=0x0, gen=0x7ffff48402e0) at Objects/genobject.c:219
219     Objects/genobject.c: Directory not empty.
(gdb)
#7  gen_iternext (gen=0x7ffff48402e0) at Objects/genobject.c:594

pablogsal commented on Nov 23, 2022
pablogsal
on Nov 23, 2022
Member
More actions
CC: @markshannon @brandtbucher
pablogsal commented on Nov 23, 2022
pablogsal
on Nov 23, 2022
Member
More actions
I can confirm that still happens with 3.11 current.
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions
Thanks, I'll look at this today (Mark's out right now).
brandtbucher
self-assigned this
on Nov 23, 2022
graingert commented on Nov 23, 2022
graingert
on Nov 23, 2022
ContributorAuthor
More actions
I've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962
Reacted by Pradyun Gedam, Brandt Bucher, Thomas Grainger and Matt Wozniski
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions

I've now eliminated pytest from my reproducer: graingert/segfault-repro@34b5962

Oh thank God. You have no idea how helpful this is.
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions
@graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with pip install -r requirements.txt; python test_ssltransport.py.
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions
I see the segfault intermittently when running pytest, though.
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions
In the meantime, I'm using pytest to reproduce. I've gotten test_ssltransport.py down to 270 lines, removed __init__.py entirely, and gotten pyproject.toml down to just:

[tool.pytest.ini_options] filterwarnings = ["error"]
brandtbucher commented on Nov 23, 2022
brandtbucher
on Nov 23, 2022
Member
More actions
Let me know if you want me to open a PR with the simplified reproducers (or, even better, if you want me to just push them).
graingert commented on Nov 23, 2022
graingert
on Nov 23, 2022
ContributorAuthor
More actions

@graingert, how should I run the new reproducer? I'm unable to get 3.11 head to crash with pip install -r requirements.txt; python test_ssltransport.py.

You need to run it with python -W error
18 remaining items
pablogsal commented on Nov 30, 2022
pablogsal
on Nov 30, 2022
Member
More actions
I still don't undetstand why this doesn't crash:

import sys class DelRaises: def __del__(self): global sneaky sneaky = sys._getframe() def run(): _ = DelRaises() print("Running...") run() print("Crashing...")

Unfortunately, I am currently dealing with a health issue so I could not find time to deal with this in detail :(
brandtbucher commented on Nov 30, 2022
brandtbucher
on Nov 30, 2022
Member
More actions
Sorry, to hear that. :(

I was able to figure this out yesterday, but was waiting to share in the meeting. I’ll post here in case you can’t come:

I was able to get a single-threaded version to crash, too. It’s just easier to do with multiple threads since the frame object has a stale pointer into the thread’s frame stack. In our threaded example, after the thread finishes, the frame stack ceases to exist, so things can go south pretty quickly. When single-threaded, we just have a pointer into an old part of the current chunk, so things appear valid for longer if you don’t try to do much.

I’m on my phone now, but if I remember correctly, doing something like sneaky.f_back.f_locals after the run() makes the single-threaded version crash.

Basically, we should mark this frame as “incomplete” (or just unlink it, not sure yet which is easier) after checking for existing frame objects, but before clearing stuff. This restores the behavior of 3.10, which made it look like __del__ was called from run’s caller, not from run itself.

I’ll have a PR up for review this week.
Reacted by Pablo Galindo Salgado, Thomas Grainger and Łukasz Langa
pablogsal
added
release-blocker
on Dec 3, 2022
ezio-melotti
added this to Release and Deferred blockers 🚫on Dec 3, 2022
ezio-melotti
moved this to Todo in Release and Deferred blockers 🚫on Dec 3, 2022
pablogsal commented on Dec 3, 2022
pablogsal
on Dec 3, 2022
Member
More actions
Marking this as 3.11 release blocker
Reacted by Łukasz Langa
brandtbucher
added
3.11only security fixesonly security fixes

3.12only security fixesonly security fixes
on Dec 5, 2022
pablogsal
added 2 commits that reference this issue on Dec 6, 2022
GH-99729: Unlink frames before clearing them (GH-100030)
b72014c
[3.11] GH-99729: Unlink frames before clearing them (#100047)
2182a71
pablogsal commented on Dec 6, 2022
pablogsal
on Dec 6, 2022
Member
More actions
PRs merged so I am closing the issue. Thanks a lot to everyone that participated in this bug, from identifying it to fixing it. You all rock 🤘
pablogsal
removed
release-blocker
on Dec 6, 2022
pablogsal
closed this as completed on Dec 6, 2022
Repository owner moved this from Todo to Done in Release and Deferred blockers 🚫on Dec 6, 2022
brandtbucher commented on Dec 6, 2022
brandtbucher
on Dec 6, 2022
Member
More actions
I've confirmed that the 3.11 patch fixes the full original reproducer, as well.

Metadata

Assignees

Labels

Projects

Release and Deferred blockers 🚫
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Frame teardown can create frame objects #99729

Description

Crash report

Error messages

Your environment

Linked PRs

Activity

graingert commented on Nov 23, 2022

pablogsal commented on Nov 23, 2022

pablogsal commented on Nov 23, 2022

pablogsal commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

graingert commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

brandtbucher commented on Nov 23, 2022

graingert commented on Nov 23, 2022

18 remaining items

pablogsal commented on Nov 30, 2022

brandtbucher commented on Nov 30, 2022

pablogsal commented on Dec 3, 2022

pablogsal commented on Dec 6, 2022

brandtbucher commented on Dec 6, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions