GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-123748: Add conditional compilation rules to allow HACL SIMD256 to be compiled on universal by freakboy3742 · Pull Request #123989 · python/cpython

Conversation

@freakboy3742

@freakboy3742 freakboy3742 commented Sep 12, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Add modifications to the compilation of hashlib to restore the ability to generate a universal build.

#119316 added an implementation of Blake2 to hashlib. That implementation compiles fine on single architecture macOS builds (both x86_64 and ARM64, as verified by CI); but universal2 builds generate a compilation error because the SIMD256 algorithm can be compiled on x86_64, but can't be compiled on ARM64. Since both architectures are compiled in a single pass, autoconf by itself can't detect or control how to include (or exclude) the SIMD256 module.

This PR:

  • Adds a Hacl_Hash_Blake2b_Simd256_universal2.c file that conditionally #includes Hacl_Hash_Blake2b_Simd256.c`. This keeps the Python additions isolated, allowing the original HACL sources to be used (and updated) as-is.
  • Adds Makefile and configure changes to include the _universal2 wrapper variant if compiling on universal, along with configure logging to identify when this occurs
  • Adds a conditional block to the start of blake2module.c that disables the HACL_CAN_COMPILE_SIMD256 symbol if on ARM64. On a pure ARM64 compile, this will never be executed; but in a universal build, it will.

Fixes #123748.

/cc @msprotz

  • Issue: Error compiling HACL* Blake2 support for macOS universal binaries #123748

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@freakboy3742
Add conditional compilation rules to allow HACL SIMD256 to be partial…
bcffc38
…ly ignored on universal builds.
@freakboy3742
freakboy3742 requested a review from ned-deily September 12, 2024 06:11
@freakboy3742
freakboy3742 requested review from corona10, erlend-aasland, gpshead and tiran as code owners September 12, 2024 06:11
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 12, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Sep 12, 2024
Error compiling HACL* Blake2 support for macOS universal binaries #123748
Closed
@freakboy3742 freakboy3742 added skip news build The build process and cross-build 3.14 bugs and security fixes labels Sep 12, 2024
@freakboy3742
Update SBOM manifest.
35c2466
@freakboy3742
freakboy3742 requested a review from sethmlarson as a code owner September 12, 2024 06:30
sethmlarson
sethmlarson approved these changes Sep 12, 2024
View reviewed changes

@sethmlarson sethmlarson left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SBOM changes LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@msprotz

msprotz commented Sep 12, 2024

Copy link
Copy Markdown
Contributor

The logic looks good to me, but I wonder if this ends up enabling Blake2s/128 on ARM64? Right now it's not enabled (due to a slow barrel-shifter, I think, which gives so-so performance) by default, but might end up being enabled unintentionally for universal builds.

Also, I don't know if this is specific to this work, but I tried out ./configure --with-builtin-hashlib-hashes=md5,sha1,sha2,sha3,blake2 but then when I tried to make sure built-in hashes were linked properly, I got this:

>>> import hashlib
>>> hashlib.sha256
<built-in function openssl_sha256>

(not sure this is related to the problem of universal builds, or if this is a more general issue with configure)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@freakboy3742

freakboy3742 commented Sep 13, 2024

Copy link
Copy Markdown
Contributor Author

The logic looks good to me, but I wonder if this ends up enabling Blake2s/128 on ARM64? Right now it's not enabled (due to a slow barrel-shifter, I think, which gives so-so performance) by default, but might end up being enabled unintentionally for universal builds.

It will - unlike SIMD256.c, SIMD128.c can be compiled on ARM64. However, given the existing configuration, it wouldn't be included in an ARM64-only build. I'll push an update to do a similar "#include and #undef" trick for the SIMD128 implementation so that we retain parity with single-architecture builds.

Also, I don't know if this is specific to this work, but I tried out ./configure --with-builtin-hashlib-hashes=md5,sha1,sha2,sha3,blake2 but then when I tried to make sure built-in hashes were linked properly, I got this:

>>> import hashlib
>>> hashlib.sha256
<built-in function openssl_sha256>

(not sure this is related to the problem of universal builds, or if this is a more general issue with configure)

This is the same behavior I see in the official universal build of Python 3.13.0rc2 running on macOS. The same is true of other older Pythons that I've tested back to 3.8, so it doesn't appear to be specific to this patch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@freakboy3742
Disable SIMD128 on the ARM64 slice of universal2 builds.
a53ce5e
ned-deily
ned-deily approved these changes Sep 13, 2024
View reviewed changes

@ned-deily ned-deily left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that universal2 and single-arch builds now compile without error. Thanks for tracking this down and following up.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 13, 2024
@freakboy3742

freakboy3742 commented Sep 16, 2024

Copy link
Copy Markdown
Contributor Author

@msprotz Are you happy with the current state of this patch?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead
gpshead approved these changes Sep 16, 2024
View reviewed changes

@gpshead gpshead left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me, thanks for untangling this one!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
msprotz
msprotz approved these changes Sep 16, 2024
View reviewed changes

@msprotz msprotz left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thank you for figuring out this tricky build issue -- much appreciated!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@freakboy3742
freakboy3742 merged commit ef530ce into python:main Sep 16, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Sep 16, 2024
@freakboy3742
freakboy3742 deleted the universal-hacl branch September 16, 2024 04:23
savannahostrowski pushed a commit to savannahostrowski/cpython that referenced this pull request Sep 22, 2024
@freakboy3742 @savannahostrowski
pythongh-123748: Add conditional compilation rules for HACL SIMD256 a…
b762a5b
…nd SIMD128 on macOS (python#123989)

Add conditional compilation rules to allow HACL SIMD256 and SIMD128 to be ignored on the ARM64 pass of universal2 macOS builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@ned-deily ned-deily ned-deily approved these changes
@sethmlarson sethmlarson sethmlarson approved these changes
@tiran tiran Awaiting requested review from tiran
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner
+1 more reviewer
@msprotz msprotz msprotz approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

3.14 bugs and security fixes build The build process and cross-build skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Error compiling HACL* Blake2 support for macOS universal binaries

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@freakboy3742 @msprotz @gpshead @ned-deily @sethmlarson
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.