GeezerVison is showing github.com — Simplify the interpreter's (type, val, tb) exception representation · Issue #89874 · python/cpython
Simplify the interpreter's (type, val, tb) exception representation #89874Description
| BPO |
45711 |
| Nosy |
@gvanrossum, @terryjreedy, @scoder, @vstinner, @markshannon, @brandtbucher, @da-woods, @iritkatriel |
| PRs |
bpo-45711: Use _PyErr_ClearExcState instead of setting only exc_value… #29404bpo-45711: remove unnecessary DUP_TOP and POP in exception handling #29495bpo-45711: assert that the type of exc_info is redundant #29518bpo-45711: Re-bump the magic number and update doc #29528bpo-45711: use exc_value instead of exc_type to determine if exc_info is valid. Add more assertions. #29627bpo-45711: Change exc_info related APIs to derive type and traceback from the exception instance #29780bpo-45711: [asyncio] Normalize exceptions immediately after Fetch, before they are stored as StackItem, which should be normalized #29890bpo-45711: Remove unnecessary normalization of exc_info #29922bpo-45711: Remove type and traceback from exc_info #30122bpo-46219, 46221: simplify except* implementation following exc_info changes. Move helpers to exceptions.c. Do not assume that exception groups are truthy. #30289bpo-45711: move whatsnew entries which are incorrectly listed under N… #30849 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2022-03-13.19:33:16.899>
created_at = <Date 2021-11-04.11:29:47.242>
labels = ['interpreter-core', '3.11', 'performance']
title = "Simplify the interpreter's (type, val, tb) exception representation"
updated_at = <Date 2022-03-13.19:33:16.899>
user = 'https://github.com/iritkatriel' bugs.python.org fields:
activity = <Date 2022-03-13.19:33:16.899>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-13.19:33:16.899>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2021-11-04.11:29:47.242>
creator = 'iritkatriel'
dependencies = []
files = []
hgrepos = []
issue_num = 45711
keywords = ['patch']
message_count = 31.0
messages = ['405681', '405839', '405849', '406114', '406120', '406209', '406238', '406983', '406984', '407402', '407597', '408043', '408768', '408786', '409453', '411497', '412264', '412265', '412270', '412272', '412273', '412275', '412281', '412289', '412290', '412292', '412303', '412304', '412316', '414482', '415086']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'terry.reedy', 'scoder', 'vstinner', 'Mark.Shannon', 'brandtbucher', 'da-woods', 'iritkatriel']
pr_nums = ['29404', '29495', '29518', '29528', '29627', '29780', '29890', '29922', '30122', '30289', '30849']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue45711'
versions = ['Python 3.11'] Activityiritkatriel commented on Nov 4, 2021 Exceptions are represented in the interpreter as (type, val, tb) triplets which most of the time contain redundant information (the type is the type of val and the tb is also on the exception). This complicates the code and is inefficient as opcodes that manage exceptions push and pop 3 items for each exception.
We will change the internal representation to be (1) just the exception value if it is normalised and (2) a tuple of the 3 values for the uncommon case where they are all needed.
See also faster-cpython/ideas#106. terryjreedy commented on Nov 6, 2021 Would there be any change at the Python level? iritkatriel commented on Nov 6, 2021 Initially not, neither in python nor in the c api.
It would be nice to replace PyErr_Fetch/Restore by a version that takes just an exception but that’s a long deprecation. iritkatriel commented on Nov 10, 2021 markshannon commented on Nov 10, 2021 iritkatriel commented on Nov 12, 2021 iritkatriel commented on Nov 12, 2021 iritkatriel commented on Nov 25, 2021 New changeset c456dfa by Irit Katriel in branch 'main':
bpo-45711: use exc_value instead of exc_type to determine if exc_info is valid. Add more assertions. (GH-29627)
c456dfa iritkatriel commented on Nov 25, 2021 Following the analysis/discussion on faster-cpython/ideas#106:
-
exc_info is always normalized, so we actually will never need to create a tuple of these three values (exc_curinfo, the exception raised but not yet caught can be unnormalized, but it is not pushed/popped on the stack).
-
We will reduce the interpreter's exc_info representation to just the exception instance.
-
There are two APIs that are impacted, both in non-documented edge cases:
-
sys.exc_info()[2] can currently be different from sys.exc_info()[1].__traceback__ because changes to the latter (while an except clause is executing) don't show up in the former. This is arguably a bug, we will change it so that the type and traceback are always consistent with the exception.
-
PyErr_SetExcInfo does no arg checking, and will set exc_info to an inconsistent triplet if you ask it to. However, the exc_value arg must be an exception instance so the only thing you can do is pass in nonsensical args where the type/traceback do not match the exception. This function's purpose is to save/restore exc_info. We will make it ignore the type and traceback and document that change.
iritkatriel commented on Nov 30, 2021 New changeset 8a45ca5 by Irit Katriel in branch 'main':
bpo-45711: Change exc_info related APIs to derive type and traceback from the exception instance (GH-29780)
8a45ca5 iritkatriel commented on Dec 3, 2021 New changeset 2ff758b by Irit Katriel in branch 'main':
bpo-45711: [asyncio] Normalize exceptions immediately after Fetch, before they are stored as StackItem, which should be normalized (GH-29890)
2ff758b iritkatriel commented on Dec 8, 2021 iritkatriel commented on Feb 1, 2022 That commit has significant changes in ceval.c and compile.c. They don't need to be reverted to unbreak cython. I'm working on a PR for a simpler change.
Have you tried with CYTHON_USE_EXC_INFO_STACK undefined? iritkatriel commented on Feb 1, 2022 gvanrossum commented on Feb 1, 2022 Time to insist on directly communicating with the Cython team (esp. @scoder) and broker some kind of compromise. vstinner commented on Feb 1, 2022
__Pyx_PyErr_GetTopmostException(PyThreadState *tstate)
Python provides a *private* _PyErr_GetTopmostException(tstate) function, but Cython reimplements its own function. I'm not sure why.
#74716 proposes adding PyErr_GetActiveException() function which has no parameter, but Cython __Pyx_PyErr_GetTopmostException() has a tstate parameter.
Simplified example numpy/random/_mt19937.c code:
static CYTHON_INLINE void
__Pyx__ExceptionSave(PyThreadState *tstate, PyObject **type,
PyObject **value, PyObject **tb)
{
_PyErr_StackItem *exc_info = __Pyx_PyErr_GetTopmostException(tstate);
*type = exc_info->exc_type;
*value = exc_info->exc_value;
*tb = exc_info->exc_traceback;
Py_XINCREF(*type);
Py_XINCREF(*value);
Py_XINCREF(*tb);
}
static CYTHON_INLINE void
__Pyx__ExceptionReset(PyThreadState *tstate, PyObject *type,
PyObject *value, PyObject *tb)
{
PyObject *tmp_type, *tmp_value, *tmp_tb;
_PyErr_StackItem *exc_info = tstate->exc_info;
tmp_type = exc_info->exc_type;
tmp_value = exc_info->exc_value;
tmp_tb = exc_info->exc_traceback;
exc_info->exc_type = type;
exc_info->exc_value = value;
exc_info->exc_traceback = tb;
Py_XDECREF(tmp_type);
Py_XDECREF(tmp_value);
Py_XDECREF(tmp_tb);
}Cython saves/restores the current exception of tstate. Maybe we need to provide a high-level API for that as well? iritkatriel commented on Feb 1, 2022 iritkatriel commented on Feb 1, 2022
#74716 proposes adding PyErr_GetActiveException() function which has no parameter, but Cython __Pyx_PyErr_GetTopmostException() has a tstate parameter.
I've now updated it to follow the pattern of other functions, where the is a private function that takes tstate and the public function calls it.
So it adds in Include/pyerrors.h
PyAPI_FUNC(PyObject*) PyErr_GetActiveException(void);
PyAPI_FUNC(void) PyErr_SetActiveException(PyObject *); and in Include/cpython/pyerrors.h
PyAPI_FUNC(PyObject*) _PyErr_GetActiveException(PyThreadState *);
PyAPI_FUNC(void) _PyErr_SetActiveException(PyThreadState *, PyObject *); da-woods commented on Feb 1, 2022 Just a quick comment on Cython and these changes:
Cython 0.29 can build itself on Python 3.11a4 with CFLAGS="-DCYTHON_FAST_THREAD_STATE=0 -DCYTHON_USE_EXC_INFO_STACK=0" python3.11 setup.py build_ext.
I think there's some coroutine code left that isn't fixed by those flags, but a large chunk of Cython libraries won't use coroutines and so will work fine with these flags. Hopefully that unblocks some stuff for you. vstinner commented on Mar 3, 2022 lxml does crash on the current Cython 0.29.x development branch. iritkatriel commented on Mar 13, 2022
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|