GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-45256: Remove the usage of the C stack in Python to Python calls by pablogsal · Pull Request #28488 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 21, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Remove the usage of the cstack in Python to Python calls
  • Fix gdb
  • Move depth to frame structure for easier retrieval in debugging tools
  • Refactor and simplifications

https://bugs.python.org/issue45256

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

stonebig, corona10, Jaime02, Abdur-rahmaanJ, cedric05, and lafolle reacted with thumbs up emoji ctismer, emmatyping, methane, Jaime02, johnthagen, sweeneyde, Abdur-rahmaanJ, and Kobzol reacted with hooray emoji stonebig, Jaime02, Abdur-rahmaanJ, and tonybaloney reacted with heart emoji corona10, emmatyping, Jaime02, and Abdur-rahmaanJ reacted with rocket emoji stonebig, ambv, and Jaime02 reacted with eyes emoji
All reactions
  • 👍 6 reactions
  • 🎉 8 reactions
  • ❤️ 4 reactions
  • 🚀 4 reactions
  • 👀 3 reactions
@pablogsal
pablogsal requested a review from markshannon as a code owner September 21, 2021 10:42
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Sep 21, 2021
@bedevere-bot bedevere-bot added the awaiting core review label Sep 21, 2021
@pablogsal pablogsal added skip news 🔨 test-with-buildbots Test PR w/ buildbots; report in status section and removed 🔨 test-with-buildbots Test PR w/ buildbots; report in status section labels Sep 21, 2021
@bedevere-bot

bedevere-bot commented Sep 21, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit b4149b6d80bc85c73104ca0f98ee3754cf5d485c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 21, 2021
pablogsal
pablogsal commented Sep 21, 2021
View reviewed changes
Comment thread Include/internal/pycore_frame.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal force-pushed the simple_cframe branch from b4149b6 to 415273d Compare September 21, 2021 11:03
markshannon
markshannon requested changes Sep 21, 2021
View reviewed changes

@markshannon markshannon left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. A few minor issues.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/gdb/libpython.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Outdated

@markshannon markshannon Sep 21, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nargsf and vector_nargs below are only used in asserts.
I think all the asserts and assignments to nargsf and vector_nargs could be replaced with assert(nargs == 0 || args != NULL);

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@Jshsus Jshsus Jan 29, 2023

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Op ha

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Include/internal/pycore_frame.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting changes and removed awaiting core review labels Sep 21, 2021
@bedevere-bot

bedevere-bot commented Sep 21, 2021

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Sep 21, 2021

Copy link
Copy Markdown
Member Author

@markshannon Damn, unfortunately something is going on with Windows:

D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: The command "setlocal
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: set PYTHONPATH=D:\a\cpython\cpython\Lib
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB3073: "D:\a\cpython\cpython\PCbuild\win32\python.exe" "D:\a\cpython\cpython\PC\validate_ucrtbase.py" ucrtbase" exited with code -1073741819.
D:\a\cpython\cpython\PCbuild\python.vcxproj(125,5): warning MSB4181: The "Exec" task returned false but did not log an error.
D:\a\cpython\cpython\PCbuild\regen.targets(107,5): error MSB3073: The command "D:\a\cpython\cpython\PCbuild\win32\python.exe Programs\freeze_test_frozenmain.py Programs/test_frozenmain.h" exited with code -1073741819. [D:\a\cpython\cpython\PCbuild\python.vcxproj]
    6 Warning(s)
    1 Error(s)

Time Elapsed 00:04:54.99

Apparently that is a File System error but I don't understand why this may be happening :(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Sep 21, 2021

Copy link
Copy Markdown
Member Author

@markshannon Oh, I found the source of the Windows failure: it was an existing bug in the PREDICT() macro for the code path without computed gotos. Fixed it in 50276ad8813bfd0bb0455d0e699e2ee301071f00

That was challenging to find indeed!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 21, 2021
@bedevere-bot

bedevere-bot commented Sep 21, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 50276ad8813bfd0bb0455d0e699e2ee301071f00 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 21, 2021
Fidget-Spinner
Fidget-Spinner reviewed Sep 21, 2021
View reviewed changes

@Fidget-Spinner Fidget-Spinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cool. I just have a few comments on readability. No idea why ASAN is failing tbh.

One other concern I have is that this slows down non-python function calls very slightly (with the two additional checks). I highly doubt it's measurable on pyperformance though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Outdated

@Fidget-Spinner Fidget-Spinner Sep 21, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self, no need to pop frame here because it's done on eval frame exit at exit_eval_frame.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/ceval.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/gdb/libpython.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Sep 21, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I cannot reproduce the ASAN bug. :(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal force-pushed the simple_cframe branch from 98fced8 to 21b7d29 Compare September 21, 2021 15:36
@Fidget-Spinner

Fidget-Spinner commented Sep 21, 2021

Copy link
Copy Markdown
Member

There're refleaks but I can't pinpoint where :(

python.exe -m test test_grammar test_builtin -R 3:3
0:00:00 Run tests sequentially
0:00:00 [1/2] test_grammar
beginning 6 repetitions
123456
......
test_grammar leaked [12, 12, 12] references, sum=36
0:00:01 [2/2/1] test_builtin -- test_grammar failed (reference leak)
beginning 6 repetitions
123456
......
test_builtin leaked [13, 13, 13] references, sum=39
test_builtin failed (reference leak)

== Tests result: FAILURE ==
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pablogsal added 5 commits September 21, 2021 17:30
@pablogsal
Remove the usage of the cstack in Python to Python calls
8a88fd3
Ths commit inlines calls to Python functions in the eval loop
and steals all the arguments in the call from the caller for
performance.
@pablogsal
Fix gdb
799e366
@pablogsal
Move depth to frame structure for easier retrieval in debugging tools
306e29f
@pablogsal
Refactor and simplifications
fc36be1
@pablogsal
Fix existing bug in the PREDICT() macro for the codepath without comp…
e1091e0
…uted gotos
@pablogsal
pablogsal force-pushed the simple_cframe branch from 21b7d29 to e1091e0 Compare September 21, 2021 16:31
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 29, 2021
@bedevere-bot

bedevere-bot commented Sep 29, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 0ddc46e25e4885562f7d1e962b9b30eea2b0283d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 29, 2021
@markshannon

markshannon commented Sep 29, 2021

Copy link
Copy Markdown
Member

https://wiki.python.org/moin/DebuggingWithGdb

which is quite suboptimal. Maybe I should add a new section to the official docs. I am happy to pack that into this PR but maybe we should do it separately.

What do you think @markshannon ?

I hadn't realized the docs were so sparse. Probably best to do it in another PR.
I've opened https://bugs.python.org/issue45317 so we don't forget.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Sep 29, 2021

Copy link
Copy Markdown
Member Author

@markshannon what do you think of 0ddc46e for now? I dislike this approach (is correct, but is difficult to reason about if you don't have the full picture in mind). Should we do the refactor now, should we do a smaller refactor or should we just do some celanup of 0ddc46e ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal pablogsal mentioned this pull request Sep 29, 2021
bpo-45116 Fix another performance regression on Windows #28630
Closed
@markshannon

markshannon commented Sep 30, 2021

Copy link
Copy Markdown
Member

@markshannon what do you think of 0ddc46e for now? I dislike this approach (is correct, but is difficult to reason about if you don't have the full picture in mind). Should we do the refactor now, should we do a smaller refactor or should we just do some celanup of 0ddc46e ?

Just do the minimal cleanup of 0ddc46e that you are happy with for now.
And add a comment as to why it is correct.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
Fix reference leaks (needs cleanup)
1de88f6
@pablogsal
pablogsal force-pushed the simple_cframe branch from 0ddc46e to 1de88f6 Compare October 7, 2021 09:58
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 7, 2021
@bedevere-bot

bedevere-bot commented Oct 7, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 1de88f6 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 7, 2021
@pablogsal
Merge branch 'main' into simple_cframe
f313e8c
@pablogsal

pablogsal commented Oct 7, 2021

Copy link
Copy Markdown
Member Author

@markshannon I had to fix some merge conflicts and I have done the cleanup. I am building again with the buildbot fleet but this should be ready for a first version.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 7, 2021
@bedevere-bot

bedevere-bot commented Oct 7, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit f313e8c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 7, 2021
Fidget-Spinner
Fidget-Spinner approved these changes Oct 7, 2021
View reviewed changes

@Fidget-Spinner Fidget-Spinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The C changes LGTM. Some minor formatting nits at https://github.com/python/cpython/pull/28488/files#r717573235 and below.

Huge disclaimer: I am not a Python-GDB expert! It would be better to have someone else reviewing those changes.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/ceval.c
PyObject *function = PEEK(oparg + 1);
if (Py_TYPE(function) == &PyFunction_Type) {
PyCodeObject *code = (PyCodeObject*)PyFunction_GET_CODE(function);
PyObject *locals = code->co_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function);

@Fidget-Spinner Fidget-Spinner Oct 7, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code from here onwards has waay more than 80 characters.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting changes labels Oct 7, 2021
@pablogsal

pablogsal commented Oct 7, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@Fidget-Spinner Thanks for the review!

I am not going to push anything new until the buildbots pass to not restart the long refleak builds yet again. I will fix the formatting issues in a new PR unless @markshannon wants to change something fundamental (I also plan to rename the frame->depth in another PR).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Fidget-Spinner

Fidget-Spinner commented Oct 7, 2021

Copy link
Copy Markdown
Member

I am not going to push anything new until the buildbots pass to not restart the long refleak builds yet again.

I was just about to mention that. Good call!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Fidget-Spinner
Fidget-Spinner reviewed Oct 9, 2021
View reviewed changes
Comment thread Python/ceval.c
// *valid* arguments (i.e. the ones that fit into the frame).
PyCodeObject *co = (PyCodeObject*)con->fc_code;
const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) {

@Fidget-Spinner Fidget-Spinner Oct 9, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GHA is warning:

comparison of integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long int’} and ‘long unsigned int’ [-Wsign-compare

and

'>': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
Suggested change
for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) {
for (size_t i = 0; i < Py_MIN(argcount, (size_t)total_args); i++) {

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@pablogsal
pablogsal merged commit b4903af into python:main Oct 9, 2021
@pablogsal
pablogsal deleted the simple_cframe branch October 9, 2021 15:51
@bedevere-bot bedevere-bot removed the awaiting merge label Oct 9, 2021
@pablogsal

pablogsal commented Oct 9, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Cleanups happening here: #28836

@markshannon After that is merged, I will create a PR to address the counter to convert it to a "entry frame" flag.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Oct 11, 2021

Copy link
Copy Markdown
Member

This is really cool, nice work @pablogsal ;-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal pablogsal mentioned this pull request Apr 10, 2022
Remove the usage of the C stack in Python to Python calls #89419
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brettcannon brettcannon brettcannon left review comments
@markshannon markshannon markshannon requested changes
@Fidget-Spinner Fidget-Spinner Fidget-Spinner approved these changes
@sweeneyde sweeneyde sweeneyde left review comments
+2 more reviewers
@ctismer ctismer ctismer left review comments
@Jshsus Jshsus Jshsus left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@pablogsal @bedevere-bot @Fidget-Spinner @markshannon @sweeneyde @vstinner @brettcannon @ctismer @Jshsus @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.