GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-29882: Fix portability bug introduced in GH-30774 by mdickinson · Pull Request #30794 · python/cpython

Conversation

@mdickinson

@mdickinson mdickinson commented Jan 22, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This PR fixes a portability bug in _Py_popcount32 that was introduced in GH-30774, and adds a comment explaining why the final line of the function is delicate.

Prior discussions:

  • bpo-29882: Add an efficient popcount method for integers #771 (comment)
  • bpo-29882: Add _Py_popcount32() function #20518 (comment)
  • bpo-29882: _Py_popcount32() doesn't need 64x64 multiply #30774 (comment)

https://bugs.python.org/issue29882

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mdickinson
bpo-29882: Fix portability bug introduced in pythonGH-30774
ff5ba30
@mdickinson mdickinson added the skip news label Jan 22, 2022
@mdickinson
mdickinson requested a review from vstinner January 22, 2022 15:26
@mdickinson mdickinson mentioned this pull request Jan 22, 2022
bpo-29882: _Py_popcount32() doesn't need 64x64 multiply #30774
Merged
@mdickinson mdickinson added skip issue and removed skip issue labels Jan 22, 2022
@vstinner

vstinner commented Jan 22, 2022

Copy link
Copy Markdown
Member

Can you please add a test for 2**28 + 1 in _testinternalcapi.test_popcount()?

ref: #30774 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mdickinson

mdickinson commented Jan 22, 2022

Copy link
Copy Markdown
Member Author

Can you please add a test for 2**28 + 1 in _testinternalcapi.test_popcount()?

Sure, will do. Though note that there's really nothing at all special about that value: any input value larger than 255 will give the wrong result under the code currently in main, on a machine with 64-bit int.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mdickinson
Add test for the value 2**28 + 1
58659fa
vstinner
vstinner approved these changes Jan 22, 2022
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding (uint32_t) cast and the added test LGTM. I didn't review the long comment ;-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added the awaiting merge label Jan 22, 2022
vstinner
vstinner reviewed Jan 22, 2022
View reviewed changes
Comment thread Include/internal/pycore_bitutils.h
// Binary: 0000 1111 0000 1111 ...
const uint32_t M4 = 0x0F0F0F0F;
// 256**4 + 256**3 + 256**2 + 256**1
const uint32_t SUM = 0x01010101;

@vstinner vstinner Jan 22, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why removing this constant? I added it to not have to use the macro to get an uint32_t literal number. UINT32_C() if I recall correctly.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mdickinson mdickinson Jan 23, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vstinner This is already explained in the long comment that you didn't review. :-)

The problem here isn't the constant; it's the type declaration.

For the multiplication in the last line of the function to be portable, we need at least one of the unsigned operands in that multiplication to not be promoted to int. An inline constant 0x01010101U satisfies that criterion: by C99 §6.4.4.1, together with C's guarantees about the minimum precision of long, it has type either unsigned int or unsigned long. A uint32_t constant with the same value does not satisfy that criterion, for reasons already explained.

And there isn't a type declaration that works here. I just explained why uint32_t won't work. If we declare SUM as unsigned int instead of uint32_t, we have portability issues on machines where int isn't large enough to represent the value 0x01010101. If we declare it as unsigned long, we're back to doing a 64-bit-by-64-bit multiply on almost all current Linux and macOS boxes.

If you really want to keep the constant, another option is to leave this line exactly as-is and change the multiplication in the last line to x * (SUM + 0U); that + 0U effectively forces the second multiplicand to have type with rank greater than or equal to that of int, making it immune to further integer promotion.

But as Tim observed in the #30774 discussion, none of this prevents us from potentially doing a 512-bit-by-512-bit multiply on a box that has 512-bit integers. But short of relying on compiler-specific intrinsics, that's inescapable anyway: standard C simply isn't capable of doing arithmetic on anything smaller than an int.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
tim-one
tim-one approved these changes Jan 22, 2022
View reviewed changes

@tim-one tim-one left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mdickinson
mdickinson merged commit 83a0ef2 into python:main Jan 23, 2022
@bedevere-bot bedevere-bot removed the awaiting merge label Jan 23, 2022
@bedevere-bot

bedevere-bot commented Jan 23, 2022

Copy link
Copy Markdown

@mdickinson: Please replace # with GH- in the commit message next time. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mdickinson
mdickinson deleted the fix-popcount-portability branch January 23, 2022 09:59
@niklasf niklasf mannequin mentioned this pull request Nov 13, 2022
Add an efficient popcount method for integers #74068
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
+1 more reviewer
@tim-one tim-one tim-one approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@mdickinson @vstinner @bedevere-bot @tim-one
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.