GeezerVison is showing github.com — Public C API for accessing code object fields · Issue #94936 · python/cpython
Public C API for accessing code object fields #94936DescriptionA user/maintainer of a popular library documented their use case for accessing fields in the code object. In 3.11 the fields in C are gone but still available as properties in Python. Accessing these properties can be made faster by exposing their functions in the C API. The current option is PyObject_GetAttrString(code, "attr") which is much slower.
https://discuss.python.org/t/getting-the-class-name-of-a-code-frame-object-in-cpython-3-11-c-api/17396
A similar issue is #92154.
I am marking this as a release blocker @pablogsal as it would be mucho bueno if we could get this before rc1. Activitypablogsal commented on Jul 17, 2022 Mucho bueno indeed :)
I am willing to consider it but how many new APIs are we talking about? Also, we should double check and agree that exposing these fields is not going to backfire on us if we decide to change the implementation or add new stuff that will invalidate the meaning of the current fields. Maybe we can consider adding them with prefixed underscores (probably a bad idea)? pablogsal commented on Jul 17, 2022 Fidget-Spinner commented on Jul 17, 2022 3 getter functions. I've thought about whether they will restrict our code. Thing is we already are restricted because all three are already exposed as Python properties. So we have no choice but to maintain backwards compatiblity, even without the new C API changes. markshannon commented on Jul 19, 2022 PyObject_GetAttrString(code, "attr") may be slower, but I doubt that PyObject_GetAttr(code, attr) is much slower given that most of the attributes of code objects are computed.
Calling PyObject_GetAttr(code, co_varnames) requires that the co_varnames tuple is created. Dispatching is not likely to be any slower than creating the tuple.
If performance is the issue, we should offer a faster API (for 3.12)
Something like:
PyObject *PyCode_GetLocalName(int n)
We should also offer the same API in Python. markshannon commented on Jul 19, 2022 Not only that, but co_nlocals and co_varnames don't make much sense.
def f(a, b):
c = a + b
def d():
return c
return ddef g(a, b):
c = a + b
def d():
return a + b
return d>>> f.__code__.co_nlocals
3
>>> g.__code__.co_nlocals
4
>>> f.__code__.co_varnames
('a', 'b', 'd')
>>> g.__code__.co_varnames
('a', 'b', 'c', 'd')
Fidget-Spinner commented on Jul 20, 2022
If performance is the issue, we should offer a faster API (for 3.12) Something like: PyObject *PyCode_GetLocalName(int n)
We should also offer the same API in Python.
Yeah I planned to lazily create the introspection information and cache them there (kind of like how debug information in frames are lazily created). However, that missed that for 3.11 so it should go in 3.12. Something like
struct PyCodeObject {
...
PyCodeIntrospectInformation *introspect_information;
}
struct debug information {
PyObject *co_code;
PyObject *co_varnames;
PyObject *co_freevars;
...
}
gvanrossum commented on Jul 20, 2022 Given that the user requesting this is writing a profiler and needs access to the frame as well, why not just recommend that they include internal headers and call _PyCode_GetVarnames(), _PyCode_GetCellvars and _PyCode_GetFreevars directly? Looking at #95008 (presumably a tentative fix?) it would seem that all the required functions already exist but aren't public any more. That seems prudent since there are some concerns about the future-preparedness of these functions anyway.
In the past I think we would never have considered such a late request for new APIs a blocker. Fidget-Spinner commented on Jul 22, 2022 OK. I'll remove the blocker. However, I think we should consider adding a performant version of these APIs in 3.12. pablogsal commented on Jul 23, 2022 After reading the discussion, I concur with Guido. I am removing the 3.11 label. This feature can only be considered from 3.12 onwards. vstinner commented on Aug 3, 2022 Fidget-Spinner commented on Aug 4, 2022 gvanrossum commented on Aug 4, 2022
Docs TODO.
What more docs do you need? Fidget-Spinner commented on Aug 4, 2022
Docs TODO.
What more docs do you need?
Woops I forgot that I already documented these. Sorry 😆 .
I'll leave this issue open for the next thing I plan to do in 3.12:
I want a better/more performant implementation for 3.12. It will cache the debug info in a lazy debug struct much like how we do it for PyInterpreterFrame and PyFrameObject now.
vstinner commented on Aug 5, 2022 IMO the problem of migrating existing C extensions to Python 3.11 is now solved. What's New in Python 3.11:
PyCodeObject no longer has the co_code, co_varnames, co_cellvars and co_freevars fields. Instead, use PyCode_GetCode(), PyCode_GetVarnames(), PyCode_GetCellvars() and PyCode_GetFreevars() respectively to access them via the C API. (Contributed by Brandt Bucher in bpo-46841 and Ken Jin in gh-92154 and gh-94936.)
I close the issue.
I'll leave this issue open for the next thing I plan to do in 3.12
Please open a separated issue for that.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|