GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-99108: Initial import of HACL-SHA3 into Python by msprotz · Pull Request #103597 · python/cpython

Conversation

@msprotz

@msprotz msprotz commented Apr 17, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I have a PR that's close to landing upstream (hacl-star/hacl-star#803). If there are any comments here, I'll be happy to tweak the PR upstream.

This replaces the in-tree SHA3 implementation for hashlib with a verified version from the HACL* project.

A few remarks:

  • just like the original implementation, the HACL implementation uses a single API for all six Keccak variants
  • I did not understand why there was a lock in sha3module.c, along with critical sections (ENTER_HASHLIB, etc.) when none of the other hashing modules have them -- I have removed this part

@gpshead please let me know what you think! I'm around, and can help make sure this gets into 3.12

  • Issue: Replace built-in hashlib with verified implementations from HACL* #99108

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Eclips4, thinkwelltwd, and erlend-aasland reacted with rocket emoji
All reactions
  • 🚀 3 reactions
msprotz added 3 commits April 17, 2023 13:29
@msprotz
Initial import of HACL-SHA3 into Python
e2b15cc
@msprotz
News entry
4d5e5c8
@msprotz
Namespacing
20bce97
@msprotz
msprotz requested review from a team and tiran as code owners April 17, 2023 21:49
@bedevere-bot bedevere-bot mentioned this pull request Apr 17, 2023
Replace built-in hashlib with verified implementations from HACL* #99108
Closed
@bedevere-bot bedevere-bot added the awaiting review label Apr 17, 2023
msprotz added 5 commits April 17, 2023 14:54
@msprotz
Missing file
be1a46b
@msprotz
Missing file
21453b8
@msprotz
make regen after I fixed up a comment
fe87970
@msprotz
Fix C analyzer
48df880
@msprotz
Remove one superfluous file following upstream fix
9f5b5a6
@gpshead gpshead self-assigned this Apr 19, 2023
@msprotz
Refresh from upstream
95f4c40
@msprotz

msprotz commented May 3, 2023

Copy link
Copy Markdown
Contributor Author

I've refreshed this PR with a newer upstream.

  • Eliminates an unused variable.
  • Switches to a saner API of finish (SHA3) and squeeze (extra length argument, only for Shake).

@gpshead the upstream PR will land soon, let me know if there's anything I can do to help make sure this gets into 3.12!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@arhadthedev arhadthedev added stdlib Standard Library Python modules in the Lib/ directory topic-SSL labels May 3, 2023
@msprotz

msprotz commented May 5, 2023

Copy link
Copy Markdown
Contributor Author

I've landed the upstream PR, meaning that this is now pulling from HACL*'s main branch. I see from https://peps.python.org/pep-0693/ that the first 3.12 beta is in just a few days. I'll keep an eye on this PR in case any change or fixup is needed. Thanks!

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead
ReSTify the NEWS entry.
b15b524
@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 5, 2023
@bedevere-bot

bedevere-bot commented May 5, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit b15b524 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 5, 2023
@gpshead

gpshead commented May 5, 2023

Copy link
Copy Markdown
Member

At first glance I believe this looks good, I'll get it in. I'm running it through the buildbots to see if anything unusual build-wise falls out to be addressed. (expect some flaky infrastructure or pre-existing issue noise among those results as well)

msprotz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@msprotz
Merge remote-tracking branch 'upstream/main' into protz_sha3
b473f8c
@msprotz

msprotz commented May 5, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Thanks, so far only the refleaks test seems relevant -- it says hashlib leaked. Is there any way to get a trace of the allocations that weren't freed? Nothing jumps out at me looking at the code right now, and I don't believe anything was done differently compared to the other hashes (for which there were no leaks issues).

EDIT: used OSX's leaks tool in conjunction with MallocStackLogging, but I'm getting allocation traces from within the interpreter, which isn't super useful. I'll keep trying to figure this out, but of course any pointers are appreciated. Thanks.

EDIT2: tried tracing the calls to malloc performed within HACL, but the addresses don't seem to match what leaks tells me has leaked... maybe the leaked objects are Python objects?

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead and others added 3 commits May 5, 2023 19:44
@gpshead
consistent code style tweak
48cf363
@msprotz
Forgot to release local buffer
4b58915
@msprotz
Merge branch 'protz_sha3' of pro.github.com:msprotz/cpython into prot…
43f4ba7
…z_sha3
@msprotz

msprotz commented May 6, 2023

Copy link
Copy Markdown
Contributor Author

I merged master in and I think I found the missing PyBuffer_Release. Can you run buildbot again?

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2023
@bedevere-bot

bedevere-bot commented May 6, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 43f4ba7 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2023
@msprotz

msprotz commented May 6, 2023

Copy link
Copy Markdown
Contributor Author

This is the only error that shows up (WASI build):

ERROR: test_with_segments (test.test_pathlib.PathSubclassTest.test_with_segments)
ERROR: test_with_segments (test.test_pathlib.PathTest.test_with_segments)
ERROR: test_with_segments (test.test_pathlib.PosixPathTest.test_with_segments)

This doesn't seem to be related, does it?

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner
AA-Turner reviewed May 6, 2023
View reviewed changes
Comment thread PCbuild/pythoncore.vcxproj Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread PCbuild/pythoncore.vcxproj Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread PCbuild/pythoncore.vcxproj.filters Resolved

Uh oh!

There was an error while loading. Please reload this page.

@msprotz
Reorder files in VCBuild
9a0613e
@msprotz

msprotz commented May 6, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@AA-Turner I pushed a commit to enforce better ordering.

I'm not sure how the list is ordered, though: from _abc.c to _weakref.c are files starting with an underscore, correctly sorted. Then, there is a mix of underscore-prefixed files and non-underscore-prefixed files, sorted alphabetically, after removing the initial underscore: arraymodule.c, ..., _operator.c, posixmodule.c, and so on.

I did my best to try to make the list look sensible. I hope this addresses your comments, let me know if it doesn't. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented May 6, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

@zooba would likely be the best to comment on what should be the case, though your changes seem alright to me -- thank you.

A

msprotz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@msprotz

msprotz commented May 7, 2023

Copy link
Copy Markdown
Contributor Author

Great, thanks. I'm also happy to do a followup PR to fix things up after this one lands, if need be.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead
gpshead approved these changes May 8, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels May 8, 2023
@gpshead
gpshead merged commit 15665d8 into python:main May 8, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 8, 2023
@msprotz
msprotz deleted the protz_sha3 branch May 8, 2023 05:04
asottile
asottile reviewed May 8, 2023
View reviewed changes
Comment thread Modules/Setup Resolved

Uh oh!

There was an error while loading. Please reload this page.

jbower-fb pushed a commit to jbower-fb/cpython that referenced this pull request May 8, 2023
@msprotz @gpshead @jbower-fb
pythongh-99108: Replace SHA3 implementation HACL* version (python#103597
82a37a4
)

Replaces our built-in SHA3 implementation with a verified one from the HACL* project.

This implementation is used when OpenSSL does not provide SHA3 or is not present.

3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.

---------

Co-authored-by: Gregory P. Smith <[email protected]>
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
@carljm
Merge branch 'main' into inlinecomp2
76077cd
* main: (47 commits)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  pythongh-104273: Remove redundant len() calls in argparse function (python#104274)
  pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)
  pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)
  pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)
  pythongh-103650: Fix perf maps address format (python#103651)
  pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 9, 2023
@carljm
Merge branch 'main' into fixsendfamily
a614216
* main: (29 commits)
  pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)
  pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)
  pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320)
  require-pr-label.yml: Add missing "permissions:" (python#104309)
  pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)
  pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)
  pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)
  pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)
  pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)
  pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)
  pythonGH-104284: Fix documentation gettext build (python#104296)
  pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)
  pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)
  pythongh-99108: fix typo in Modules/Setup (python#104293)
  pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)
  pythongh-103193: Improve `getattr_static` test coverage (python#104286)
  Trim trailing whitespace and test on CI (python#104275)
  pythongh-102500: Remove mention of bytes shorthand (python#104281)
  pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)
  pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)
  ...
@hauntsaninja hauntsaninja mentioned this pull request Jun 27, 2023
gh-106123: Modules/_sha3 appears to no longer be necessary #106124
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AA-Turner AA-Turner AA-Turner left review comments
@gpshead gpshead gpshead approved these changes
@tiran tiran Awaiting requested review from tiran
+1 more reviewer
@asottile asottile asottile left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

stdlib Standard Library Python modules in the Lib/ directory topic-SSL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@msprotz @bedevere-bot @gpshead @AA-Turner @asottile @arhadthedev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.