GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-108494: Argument Clinic: inline parsing code for positional-only parameters in the limited C API by serhiy-storchaka · Pull Request #108622 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Aug 29, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

It fixes some code which uses private converters, like _PyLong_UnsignedShort_Converter. Inlined code only uses the limited C API.

But it still includes internal headers. It is difficult to not include them, because they are needed when the parsing code is not inlined, and this depends not only on the specified converter, but on other converters and other used features.

There is a behavior change: converters like _PyLong_UnsignedShort_Converter raise ValueError for negative integer, but the inlined code raise OverflowError. It is very difficult to detect negative integer in the limited C API. It can be fixed by changing exception in PyLong_AsUnsignedLong (see #74019).

I also reorganized the code for keyword parameters, so it will be easier to enable inlining once we add a public version of _PyArg_UnpackKeywords in the limited C API, which is the only stopper.

All tests (except one) are passed with using the limited C API (if possible).

  • Issue: Add support for the limited C API to Argument Clinic #108494

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland reacted with eyes emoji
All reactions
  • 👀 1 reaction
@serhiy-storchaka serhiy-storchaka added skip news topic-argument-clinic labels Aug 29, 2023
@serhiy-storchaka
serhiy-storchaka requested a review from vstinner August 29, 2023 10:04
@serhiy-storchaka
serhiy-storchaka requested review from a team, AlexWaygood, erlend-aasland and gpshead as code owners August 29, 2023 10:04
@bedevere-bot bedevere-bot mentioned this pull request Aug 29, 2023
Add support for the limited C API to Argument Clinic #108494
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Aug 29, 2023
@serhiy-storchaka
serhiy-storchaka removed request for a team and gpshead August 29, 2023 10:05
@serhiy-storchaka
pythongh-108494: Argument Clinic: inline parsing code in the limited …
31238a6
…C API
@serhiy-storchaka
serhiy-storchaka force-pushed the clinic-limited-capi-inline branch from 1cef242 to 31238a6 Compare August 29, 2023 10:25
vstinner
vstinner reviewed Aug 30, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR avoids private _PyArg_BadArgument() and _PyArg_CheckPositional() functions in the limited C API, and generate similar code for that. What about adding these functions to the limited C API instead?

The advantage of your approach is that we can target limited C API older than version 3.13 which is appealing. But is it worth it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/clinic/clinic.py
pl = '' if min_pos == 1 else 's'
parser_code.append(normalize_snippet(f"""
if ({nargs} != {min_pos}) {{{{
PyErr_Format(PyExc_TypeError, "{{name}} expected {min_pos} argument{pl}, got %zd", {nargs});

@vstinner vstinner Aug 30, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generate many static C strings for similar error messages. Would it make sense to pass name and min_pos (and "s" for plural) as argument to PyErr_Format(), and have a unique format string? It would be faster but make the Python binary smaller and use less memory.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Aug 31, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_PyArg_BadArgument() is has 4 parameters, but it is not generic enough yet. It formats an error message in the style of some error messages formatted in PyArg_ParseTuple() and PyArg_ParseTupleAndKeywords(), but even these functions has cases which cannot be covered by _PyArg_BadArgument() ("must be %d-item sequence", "must be sequence of length %d"), and there are many other forms of error messages for wrong type of argument in the rest of code. If _PyArg_BadArgument() be general enough and used in many other places outside of Argument Clinic generated code, it will bi worth to add it to the public API.

On other hand, PyErr_Format() is extremely general and flexible variant of _PyArg_BadArgument().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Tools/clinic/clinic.py
indent=4))
else:
parser_code = [normalize_snippet(f"""
if (!_PyArg_CheckPositional("{{name}}", {nargs}, {min_pos}, {max_args})) {{{{

@vstinner vstinner Aug 30, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about moving _PyArg_CheckPositional() to the limited C API?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Aug 31, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can be done, together with _PyArg_NoKeywords(), _PyArg_NoPositional() and _PyArg_NoKwnames().

But the main problem is handling keyword arguments, so I want to add new API only when we see the whole image.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland and gpshead reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
serhiy-storchaka
serhiy-storchaka commented Aug 31, 2023
View reviewed changes
Comment thread Tools/clinic/clinic.py

elif (not requires_defining_class and pos_only == len(parameters) and
not pseudo_args and limited_capi):
# positional-only for the limited C API

@serhiy-storchaka serhiy-storchaka Aug 31, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is no longer needed, because there is other PyArg_ParseTuple() below which is used if inlining the parsing code failed. PyArg_UnpackTuple() no longer used, because inlining is always successful in this simple case.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Tools/clinic/clinic.py
nargs = f"Py_MIN(nargs, {max_pos})" if max_pos else "0"

if limited_capi:
# positional-or-keyword arguments

@serhiy-storchaka serhiy-storchaka Aug 31, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was moved down, to a fallback code used if inlining failed. For now, it always failed for limited_capi, because there is no public variant of _PyArg_UnpackKeywords.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland and gpshead reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

@erlend-aasland erlend-aasland Aug 31, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanatory review comments.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with heart emoji
All reactions
  • ❤️ 1 reaction
erlend-aasland
erlend-aasland reviewed Aug 31, 2023
View reviewed changes
Comment thread Tools/clinic/clinic.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland

erlend-aasland commented Aug 31, 2023

Copy link
Copy Markdown
Contributor

I don't have the bandwith to review the generated C param parsing code. I guess Victor does a better job of reviewing that, than me. If Alex is fine with the clinic.py changes, you are good to go.

BTW, I really like the introduction of the fastcall variables; it makes the code more readable and understandable.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland approved these changes Aug 31, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Aug 31, 2023
AlexWaygood
AlexWaygood approved these changes Aug 31, 2023
View reviewed changes

@AlexWaygood AlexWaygood left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto to what Erlend said: I can't really review the generated C code, but I approve of the concept and the clinic.py changes look good to me

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner
vstinner approved these changes Sep 1, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The overall change LGTM. clinic.py coding style makes reviews really complicated :-( I prefer testing fastcall than new_or_init, that's a nice improvment.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
gpshead
gpshead approved these changes Sep 1, 2023
View reviewed changes

@gpshead gpshead left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work! if there are bugs lurking in here (it is hard to visually spot everything in this kind of large clinic.py change) i'm sure they'll show themselves in generated code behaviors that can be fixed up later.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
serhiy-storchaka added 2 commits September 3, 2023 15:58
@serhiy-storchaka
Merge branch 'main' into clinic-limited-capi-inline
9e5f3d2
@serhiy-storchaka
Change TODO to NOTE.
b43b74c
@serhiy-storchaka
serhiy-storchaka merged commit 1796c19 into python:main Sep 3, 2023
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting merge labels Sep 3, 2023
@serhiy-storchaka
serhiy-storchaka deleted the clinic-limited-capi-inline branch September 3, 2023 14:28
@bedevere-bot

bedevere-bot commented Sep 3, 2023

Copy link
Copy Markdown

There's a new commit after the PR has been approved.

@gpshead, @vstinner, @erlend-aasland, @AlexWaygood: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot
bedevere-bot requested review from AlexWaygood, erlend-aasland, gpshead and vstinner September 3, 2023 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexWaygood AlexWaygood Awaiting requested review from AlexWaygood
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review skip news topic-argument-clinic

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@serhiy-storchaka @erlend-aasland @bedevere-bot @gpshead @vstinner @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.