GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-99761: add invalid_index macro by eendebakpt · Pull Request #99762 · python/cpython

Conversation

@eendebakpt

@eendebakpt eendebakpt commented Nov 24, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

In listobject.c there is an optimization to check whether an index is valid (e.g. 0 <= index < N) using a single comparison. The cast-to-unsigned optimization is used in current main in 3 locations: tupleobject.c, _collectionsmodule.c and bytesobject.c.

It is a micro optimization, but the code generated is different than the plain i < 0 || i >= Py_SIZE(a). This PR tries to:

i) apply the optimization to more locations
ii) make the code more consistent

By replacing index checks with a single method _Py_is_valid_index that includes the optimization we have consistency in the code and have the optimized check for all index checks.

  • Issue: Consistent use of valid index check #99761

Notes:

  • The new method is private and is static inline (https://peps.python.org/pep-0670/)
  • In bytecodes.c the optimized expression is used in three places. E.g.
DEOPT_IF(((size_t)signed_magnitude) > 1, BINARY_SUBSCR);

With the _Py_is_valid_index method it looks like

DEOPT_IF(!_Py_is_valid_index(signed_magnitude, 2), BINARY_SUBSCR);

which is in my opinion not very readable. For these cases a separate PR was created: #100064

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@eendebakpt
add invalid_index
0d1b08a
@eendebakpt
eendebakpt requested review from markshannon and rhettinger as code owners November 24, 2022 22:11
@bedevere-bot bedevere-bot mentioned this pull request Nov 24, 2022
Consistent use of valid index check #99761
Closed
@bedevere-bot bedevere-bot added the awaiting review label Nov 24, 2022
@eendebakpt
eendebakpt marked this pull request as draft November 24, 2022 22:12
@sweeneyde

sweeneyde commented Nov 24, 2022

Copy link
Copy Markdown
Member

I think this is generally a good idea, but I'm not sure this should go in the public API. If it does, its name should probably begin with 'Py...'. See https://devguide.python.org/developer-workflow/c-api/ and https://peps.python.org/pep-0007/#naming-conventions .

cc @vstinner

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt and others added 5 commits November 25, 2022 09:55
@eendebakpt
replace valid_index
be6143f
@blurb-it
📜🤖 Added by blurb_it.
bcaec88
@eendebakpt
make regen-all
6dd6fec
@eendebakpt
Merge branch 'list_tuple' of github.com:eendebakpt/cpython into list_…
7a35197
…tuple
@eendebakpt
Rename 2022-11-25-09-34-57.gh-issue-9761.sRqH37.rst to 2022-11-25-09-…
1e4448f
…34-57.gh-issue-99761.sRqH37.rst
@eendebakpt

eendebakpt commented Nov 25, 2022

Copy link
Copy Markdown
Contributor Author

I think this is generally a good idea, but I'm not sure this should go in the public API. If it does, its name should probably begin with 'Py...'. See https://devguide.python.org/developer-workflow/c-api/ and https://peps.python.org/pep-0007/#naming-conventions .

cc @vstinner

Agreed. What would be a good location for the private version? One of the Include/internal/python_xxxx files?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eendebakpt
eendebakpt force-pushed the list_tuple branch from 37a83e2 to d85a883 Compare November 25, 2022 19:14
@eendebakpt
add method for int check
1d58ef0
@eendebakpt
eendebakpt force-pushed the list_tuple branch from d85a883 to 1d58ef0 Compare November 25, 2022 19:15
@rhettinger

rhettinger commented Nov 26, 2022

Copy link
Copy Markdown
Contributor
  • Keep the code as valid_index rather than invalid_index so that we don't even up with double negatives.

  • Please leave the collections.deque code alone. The author (me) aspires to keep this code not tightly coupled to the rest of the C API. The code is currently clean an doesn't need to be changed.

  • Are you certain that inline functions always get inlined? We know that macros do always get inlined. The valid_index macro currently gets used in performance critical code.

  • As Dennis says, this should not be part of the public API. It is a performance trick that happens to currently be useful. As compilers get better, this would be done automatically and we would prefer the simple inline code rather than the extra layer of abstraction.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt added 8 commits November 26, 2022 22:11
@eendebakpt
change invalid_index to valid_index
76ecde6
@eendebakpt
change invalid_index to valid_index
c356130
@eendebakpt
revert changes to _collectionsmodule.c on request of the author
329d6ed
@eendebakpt
make valid_index private
bbdb3b5
@eendebakpt
change static inline to macro
45d4abf
@eendebakpt
typo
e78d129
@eendebakpt
unto changes to _collectionsmodule.c
5cc1dfc
@eendebakpt
Merge branch 'main' into list_tuple
a112a05
@eendebakpt

eendebakpt commented Nov 26, 2022

Copy link
Copy Markdown
Contributor Author
  • Keep the code as valid_index rather than invalid_index so that we don't even up with double negatives.

Everywhere in the code the usage is !valid_index, which is why I preferred invalid_index. I renamed to valid_index again.

  • Please leave the collections.deque code alone. The author (me) aspires to keep this code not tightly coupled to the rest of
    the C API. The code is currently clean an doesn't need to be changed.

Done.

  • Are you certain that inline functions always get inlined? We know that macros do always get inlined. The valid_index macro currently gets used in performance critical code.

With inline functions I think we can never be certain the code gets inlined for all compilers and settings. I changed the static inline to a macro.

  • As Dennis says, this should not be part of the public API. It is a performance trick that happens to currently be useful. As compilers get better, this would be done automatically and we would prefer the simple inline code rather than the extra layer of abstraction.

The valid_index is now private (part of Includes/internal/pycore_abstract.h). If there is a more suitable location we can move it.

@rhettinger

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eendebakpt
eendebakpt marked this pull request as ready for review November 26, 2022 22:12
@vstinner

vstinner commented Nov 28, 2022

Copy link
Copy Markdown
Member

You should not use macro but a static inline functions, see:

  • https://devguide.python.org/developer-workflow/c-api/index.html
  • https://peps.python.org/pep-0670/

@rhettinger:

Are you certain that inline functions always get inlined? We know that macros do always get inlined. The valid_index macro currently gets used in performance critical code.

See PEP 670 which answers to that and we approved by the Steering Council. A static inline function must be used.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Nov 28, 2022
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very excited by a function only used for a micro-optimization, I'm fine with (i < 0 || i >= Py_SIZE(self)) test. But I'm not against this change neither.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Include/internal/pycore_abstract.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

@eendebakpt
rename to static inline int _Py_is_valid_index
d04aa18
eendebakpt added 6 commits December 26, 2022 21:59
@eendebakpt
Merge branch 'main' into list_tuple
8633a20
@eendebakpt
Merge branch 'main' into list_tuple
55e6d70
@eendebakpt
Merge branch 'main' into list_tuple
4185f3c
@eendebakpt
Merge branch 'main' into list_tuple
24f0aa0
@eendebakpt
Merge branch 'main' into list_tuple
8c1332a
@eendebakpt
Merge branch 'main' into list_tuple
8ca17f7
arhadthedev
arhadthedev reviewed Feb 20, 2023
View reviewed changes
Comment thread Include/internal/pycore_abstract.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt added 2 commits February 20, 2023 10:51
@eendebakpt
review comments
4377b07
@eendebakpt
Merge branch 'main' into list_tuple
2e871e9
@eendebakpt

eendebakpt commented Mar 6, 2023

Copy link
Copy Markdown
Contributor Author

The overall change LGTM, but please address my two remaining change requests.

@vstinner Could you have a look at the PR again?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eendebakpt added 2 commits March 19, 2023 10:40
@eendebakpt
Merge branch 'main' into list_tuple
4699151
@eendebakpt
Merge branch 'main' into list_tuple
795bf2d
@arhadthedev arhadthedev added interpreter-core (Objects, Python, Grammar, and Parser dirs) extension-modules C modules in the Modules dir skip news labels Apr 5, 2023
@eendebakpt
Merge branch 'main' into list_tuple
3ca5d8b
@eendebakpt
eendebakpt requested a review from arhadthedev April 28, 2023 19:07
@eendebakpt
Merge branch 'main' into list_tuple
fe17e85
arhadthedev
arhadthedev approved these changes May 20, 2023
View reviewed changes

@arhadthedev arhadthedev left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All earlier reviews seem to be addressed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels May 20, 2023
eendebakpt added 6 commits June 22, 2023 00:05
@eendebakpt
Merge branch 'main' into list_tuple
7cb70fc
@eendebakpt
Merge branch 'main' into list_tuple
a072101
@eendebakpt
Merge branch 'main' into list_tuple
7204d8a
@eendebakpt
Merge branch 'main' into list_tuple
977e7cc
@eendebakpt
Merge branch 'main' into list_tuple
d3b633e
@eendebakpt
Merge branch 'main' into list_tuple
cad941a
@eendebakpt eendebakpt mentioned this pull request Jun 8, 2024
gh-120176: Reduce number of compares in index range checks #120181
Closed
@serhiy-storchaka

serhiy-storchaka commented Jun 8, 2024

Copy link
Copy Markdown
Member

This idea was already discussed and rejected. See #72583.

eendebakpt reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka serhiy-storchaka closed this Jun 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arhadthedev arhadthedev arhadthedev approved these changes
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review extension-modules C modules in the Modules dir interpreter-core (Objects, Python, Grammar, and Parser dirs) skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@eendebakpt @sweeneyde @rhettinger @vstinner @serhiy-storchaka @arhadthedev @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.