GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — WIP bpo-44800: Rename `_PyInterpreterFrame` to `_Py_framedata` by ncoghlan · Pull Request #27525 · python/cpython

Conversation

@ncoghlan

@ncoghlan ncoghlan commented Aug 1, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Superseded by #31987


bpo-44590 significantly improved execution speed by delaying creation
of full Python objects for frames until they were needed for tracing
or other introspection purposes.

This follow-up commit doesn't include any further functional changes,
it just updates variable, attribute, and function names to make it
less ambiguous as to whether different sections of code are dealing
with frames as full Python objects or working directly with the new
lighter weight runtime frame data storage (C structs with no intrinsic
instance lifecycle management).

https://bugs.python.org/issue44800

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland reacted with thumbs up emoji auvipy reacted with heart emoji
All reactions
  • 👍 1 reaction
  • ❤️ 1 reaction
@ncoghlan
bpo-44800: Clearly distinguish execution & introspection frames
a133e0d
bpo-44590 significantly improved execution speed by delaying creation
of full Python objects for frames until they were needed for tracing
or other introspection purposes.

This follow-up commit doesn't include any further functions changes,
it just changes variable, attribute, and function names to make it
less ambiguous as to whether different sections of code are dealing
with the classic introspection frames (full Python objects) or the
new lighter weight execution frames (C structs with no intrinsic
instance lifecycle management).
@ncoghlan
ncoghlan requested a review from markshannon August 1, 2021 08:30
@ncoghlan
ncoghlan requested a review from 1st1 as a code owner August 1, 2021 08:30
@bedevere-bot bedevere-bot added the awaiting core review label Aug 1, 2021
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Aug 1, 2021
ncoghlan
ncoghlan commented Aug 1, 2021
View reviewed changes
Comment thread Include/cpython/frameobject.h Outdated
* executing ordinary functions), by a generator or coroutine object (for
* frames that are able to be suspended), or by their corresponding
* introspection frame (if an instrospection API has been invoked and the
* introspection frame created).

@ncoghlan ncoghlan Aug 1, 2021

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markshannon This new comment needs review to make sure I've accurately captured the way the new frame implementation works.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ncoghlan ncoghlan mentioned this pull request Aug 1, 2021
bpo-44590: Lazily allocate frame objects #27077
Merged
markshannon
markshannon requested changes Aug 9, 2021
View reviewed changes

@markshannon markshannon left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR seems to be 3 PRs in one:
1 Fixes to gdbinit
2 Changing the name of InterpreterFrame to _PyExecFrame
3 Adorning many fields with "x"

  1. Thanks for the fix. Would you put the fixes to gdbinit in its own PR, please.
    Do you have any ideas on how to add tests for this, to avoid anyone (probably me 🙂) breaking it again?

  2. Naming, the first hard problem in computer science! I think "Exec" doesn't really convey any more information than "Interpreter". Also, there should be probably be no "_Py" prefix, as that weakly implies that _PyExecFrame is a PyObject, which it is not.
    How about the old fashioned term "activation record"? The term "frame" is now so overloaded.

  3. I left the original names mostly unchanged for a reason. If the value of the attribute, e.g. f_lasti is unchanged, then it makes porting code easier if the name is unchanged. Which is why f_back becomes previous, because the value has changed. I'm sure there are some clarifications that could be made here, but could they go in a PR after we have decided on a name?

Finally, I'd avoid the term "introspection frame". From the point of view of Python code, the frame object is the frame, not just a view of it. Maybe phrase any comments in terms of the InterpreterFrame/_PyExecFrame/ActivationRecord being a storage optimization of the frame?

I believe that @pablogsal has been looking into writing up the way that the call stack now works, so he might have some thoughts on this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting changes and removed awaiting core review labels Aug 9, 2021
@bedevere-bot

bedevere-bot commented Aug 9, 2021

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ncoghlan

ncoghlan commented Aug 11, 2021

Copy link
Copy Markdown
Contributor Author

The main naming alternative that occurred to me is "frame" and "frame data".

As far as porting goes, the extra layer of indirection is likely to be the biggest hassle, and a one time porting burden for code poking around in frame internals doesn't seem like a compelling reason to tolerate ongoing ambiguity across the code base as to which kind of structure is being used.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ncoghlan

ncoghlan commented Aug 11, 2021

Copy link
Copy Markdown
Contributor Author

I posted the gist of the next iteration I'm going to try to bpo issue (including checking to see if the code still feels unambiguous if the variable names are changed while leaving the struct field names alone): https://bugs.python.org/issue44800#msg399381

For gdbinit, it's just an example file that has a few other problems with it at the moment, so I think it's OK to settle on a final field naming convention before fixing the affected pieces of it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@markshannon

markshannon commented Aug 11, 2021

Copy link
Copy Markdown
Member

As far as porting goes, the extra layer of indirection is likely to be the biggest hassle, and a one time porting burden for code poking around in frame internals doesn't seem like a compelling reason to tolerate ongoing ambiguity across the code base as to which kind of structure is being used.

Where is the ambiguity? There are no field names shared between InterpreterFrame and PyFrameObject.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ncoghlan

ncoghlan commented Aug 13, 2021

Copy link
Copy Markdown
Contributor Author

Where is the ambiguity? There are no field names shared between InterpreterFrame and PyFrameObject.

That is the ambiguity: situations where the only quick way to tell which kind of object you had was to have all the field names on each type memorised and infer the type from that, rather than from the types using different variable or field naming conventions. (Relying on the type declarations isn't good, as many of the affected functions are so long that most diffs in code reviews won't have the declaration readily accessible)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ncoghlan added 2 commits August 14, 2021 14:15
@ncoghlan
'frame' and 'frame data' replaces introspection and execution frames
d8858aa
@ncoghlan
Merge remote-tracking branch 'origin/main' into bpo-44800-rename-inte…
cd340b0
…rpreter-frames
@ncoghlan ncoghlan changed the title bpo-44800: Clearly distinguish execution & introspection frames bpo-44800: Rename InterpreterFrame to _Py_framedata Aug 14, 2021
@ncoghlan
Tweak some comments
ccf953b
ncoghlan
ncoghlan commented Aug 14, 2021
View reviewed changes
Comment thread Include/internal/pycore_framedata.h Outdated
int stackdepth; /* Depth of value stack */
int nlocalsplus;
PyFrameState state; /* What state the frame is in */
PyObject *stack[1];

@ncoghlan ncoghlan Aug 14, 2021

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file name change obscures the changes to the field names here, where the following previously had the f_ prefix:

  • globals
  • builtins
  • locals
  • code
  • lasti
  • state

To reduce the diff size, while still ensuring that the field prefix was consistently different between PyFrameObject and _Py_framedata, I went with the option of standardising the data struct on "no field prefix" (it was previously split 50/50 between that and the f_ prefix), rather than introducing a new prefix the way I did in the original PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ncoghlan ncoghlan Mar 6, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted this file rename from the PR - the internal header now remains pycore_frame.h, with both the full frame object and frame data struct field definitions in it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ncoghlan added 5 commits August 14, 2021 14:48
@ncoghlan
Another comment fix
4a097bd
@ncoghlan
Fix LLTRACE macro compile error
bd00490
@ncoghlan
Revert unintended function name changes
04aa7e8
@ncoghlan
Fix comment alignment
e9018e7
@ncoghlan
Follow proposed new naming conventions in gdb hooks
0ce41c8
@ncoghlan

ncoghlan commented Aug 14, 2021

Copy link
Copy Markdown
Contributor Author

@markshannon I have made the requested changes; please review again.

I've updated the PR title and description, since the proposal has changed based on your initial review comments.

I quite like the way the now proposed "frame" (PyFrameObject) vs "frame data" (_Py_framedata) naming convention looks in the code.

For the field prefix selection on the frame data struct, of the 12 fields, 6 didn't have a prefix at all, while 6 shared the f_ prefix with PyFrameObject. Rather than the "maximum churn" option of my original PR (which affected all 12 fields by introducing the xf_ prefix), the latest iteration instead standardises on "no field prefix" for the frame data structs.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting changes label Aug 14, 2021
@bedevere-bot

bedevere-bot commented Aug 14, 2021

Copy link
Copy Markdown

Thanks for making the requested changes!

@markshannon: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added the awaiting change review label Aug 14, 2021
@bedevere-bot
bedevere-bot requested a review from markshannon August 14, 2021 05:04
ncoghlan added 6 commits March 12, 2022 17:17
@ncoghlan
Use fdata for frame data locals and parameters
ebda1d3
@ncoghlan
frame -> fdata in ceval.c & allow compilation
269a4a0
@ncoghlan
Disambiguate f_fdata and f_frame_data
34cf023
@ncoghlan
Merge remote-tracking branch 'origin/main' into bpo-44800-rename-inte…
55d9276
…rpreter-frames
@ncoghlan
Document the currently implemented conventions
3eba918
@ncoghlan
Note the 'current_frame' exception
e8a4adf
ncoghlan
ncoghlan commented Mar 12, 2022
View reviewed changes
Comment thread Doc/whatsnew/3.11.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

ncoghlan
ncoghlan commented Mar 12, 2022
View reviewed changes
Comment thread Misc/NEWS.d/next/Core and Builtins/2021-08-01-18-18-51.bpo-44800.TCsfH3.rst
variable names ``frame`` and ``f``) and the new ``_Py_framedata`` internal
frame data storage (C structs with no intrinsic instance lifecycle management)
that is now used for code execution (function prefix ``_Py_framedata``, no
field prefix, typical variable name ``fdata``).

@ncoghlan ncoghlan Mar 12, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blurb details and date still need to be updated

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ncoghlan
ncoghlan commented Mar 12, 2022
View reviewed changes
Comment thread Objects/lnotab_notes.txt
number we save the bytecode indices where it's valid:

*instr_lb <= frame->f_lasti < *instr_ub
*instr_lb <= frame->lasti < *instr_ub

@ncoghlan ncoghlan Mar 12, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like the What's New, this snippet still isn't right, but is also wrong on main

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ncoghlan
Fix test_gdb
3d654a0
@ncoghlan

ncoghlan commented Mar 12, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

It's not currently clear why building and testing is current working locally on Fedora 35, but freezing importlib is failing when the CI goes to check that the generated files are all up to date.

Debug vs non-debug build issue (debug builds are unhappy at the moment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ncoghlan added 5 commits March 12, 2022 23:56
@ncoghlan
Fix header file include guard var
b09b114
@ncoghlan
Distinguish frame state error messages
9b51976
@ncoghlan
super() does not access C frame structs
0a3611c
@ncoghlan
new_frame -> new_fdata in frame push
08410cc
@ncoghlan
Add missing error check in PyImport_Import
c694768
@ncoghlan
ncoghlan requested review from brettcannon, encukou, ericsnowcurrently and warsaw as code owners March 12, 2022 14:32
@ncoghlan

ncoghlan commented Mar 12, 2022

Copy link
Copy Markdown
Contributor Author

Partially tracked down the debug build failure - somehow PyImport_Import is being called without the Python thread state being set up correctly first while freezing modules, and there's a PyEval_GetGlobals() call in there which isn't being correctly checked for errors (which can happen when there's no thread state set). In a release build, that error was being silently ignored, in a debug build, an assertion caught it. I've added the missing error check so that now always fails (that change should really be pulled out into its own issue report and PR...)

Still not clear how any of the name changes could have affected when the frame state gets populated, though.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ncoghlan

ncoghlan commented Mar 12, 2022

Copy link
Copy Markdown
Contributor Author

As far as I can tell, it's actually legitimate to have no active Python frame when calling PyImport_Import(), as the root_cframe struct doesn't automatically get a Python frame allocated (that doesn't happen until the Python eval loop starts running).

So while I'm still not clear on why this branch is hitting that issue when the main branch doesn't, explicitly ignoring the error seems like the right thing to do.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ncoghlan added 2 commits March 13, 2022 00:57
@ncoghlan
No Python frame seems legit for PyImport_Import()
ba87ef3
@ncoghlan
Get test_gdb passing locally
7168f7d
@ncoghlan

ncoghlan commented Mar 19, 2022

Copy link
Copy Markdown
Contributor Author

Closing this one, see #31987 for a scaled back version of the proposed change that leaves local variable and function parameter names alone.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ncoghlan ncoghlan closed this Mar 19, 2022
@ncoghlan ncoghlan mentioned this pull request Mar 19, 2022
bpo-44800: rename _PyInterpreterFrame to _Py_frame #31987
Closed
ncoghlan
ncoghlan commented Apr 3, 2022
View reviewed changes
Comment thread Python/ceval.c
if (current_frame == NULL) {
_Py_framedata *fdata = tstate->cframe->current_frame;
if (fdata == NULL) {
_PyErr_SetString(tstate, PyExc_SystemError, "cannot get globals; frame data does not exist");

@ncoghlan ncoghlan Apr 3, 2022

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this was the origin of the PyImport_ImportModule failure - in main, failing to get the locals sets an error, but failing to get the globals doesn't.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ncoghlan ncoghlan mentioned this pull request Apr 16, 2022
Code readability: rename InterpreterFrame to _Py_framedata #88963
Closed
@markshannon markshannon mentioned this pull request Apr 21, 2022
Create frame objects lazily when needed #88756
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merwok merwok merwok left review comments
@1st1 1st1 Awaiting requested review from 1st1
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@iritkatriel iritkatriel Awaiting requested review from iritkatriel
@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon is a code owner
@encukou encukou Awaiting requested review from encukou
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@warsaw warsaw Awaiting requested review from warsaw warsaw is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting change review DO-NOT-MERGE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@ncoghlan @bedevere-bot @markshannon @merwok @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.