GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-131876: extract `_hashlib` helpers into a separate directory by picnixz · Pull Request #136995 · python/cpython

Conversation

@picnixz

@picnixz picnixz commented Jul 22, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is a clean rewrite of #135341.

  • Issue: Unify recognization of message digest names for _hashlib and _hmac #131876

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@picnixz picnixz changed the title gh-135341: extract _hashlib helpers into a separate directory gh-131876: extract _hashlib helpers into a separate directory Jul 22, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Jul 22, 2025
Unify recognization of message digest names for _hashlib and _hmac #131876
Open
picnixz added 4 commits July 22, 2025 15:23
@picnixz
explode hashlib.h into multiple files
4e214fa
@picnixz
update usage of hashlib.h
854bf43
@picnixz
update build configuration
e1d233e
@picnixz
blurb
59b0cda
@picnixz
picnixz force-pushed the refactor/hashlib/extract-helpers-135341 branch from 93fdbd3 to 59b0cda Compare July 22, 2025 13:23
@picnixz picnixz added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 22, 2025
@bedevere-bot

bedevere-bot commented Jul 22, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 59b0cda 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136995%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 22, 2025
@picnixz
picnixz marked this pull request as ready for review July 25, 2025 14:00
@picnixz
picnixz requested review from a team, corona10, ericsnowcurrently, erlend-aasland, gpshead and tiran as code owners July 25, 2025 14:00
@bedevere-app bedevere-app Bot added the awaiting core review label Jul 25, 2025
@picnixz
picnixz removed request for corona10, ericsnowcurrently and erlend-aasland July 25, 2025 14:00
picnixz added 2 commits July 26, 2025 09:02
@picnixz
remove code that is currently unused
4263688
@picnixz
cleanup imports
eb817fd
gpshead
gpshead approved these changes Jul 27, 2025
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jul 27, 2025
@picnixz

picnixz commented Jul 27, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I'll first merge the fortication PR, then merge this one where the macros are rewritten are pure functions when possible. I actually wanted to check whether using an exported function vs a static inline function could change a lot the performance (if possible I don't want to make the code significantly slower because I used non-static inline functions).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead

gpshead commented Jul 28, 2025

Copy link
Copy Markdown
Member

for these code paths I doubt inline even matters. in general these days I assume the compiler will figure that out for static things, at least in pgo and lto builds.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits July 28, 2025 10:09
@picnixz
Merge remote-tracking branch 'upstream/main' into refactor/hashlib/ut…
54a10b0
…il-headers-135341
@picnixz
amend comments
06d2c87
@bedevere-app bedevere-app Bot removed the awaiting merge label Jul 28, 2025
@picnixz
remove unused rules
a1a2f1b
@picnixz

picnixz commented Jul 28, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

for these code paths I doubt inline even matters. in general these days I assume the compiler will figure that out for static things, at least in pgo and lto builds.

Actually, I was also a bit worried about the exported symbol. I also try to see if I can just live with extern and not with PyAPI_FUNC actually and if everything links correctly, then I'll be able to move the function definitions into a .c file instead.

Otherwise, I'll use static inline for both instead of using PyAPI_FUNC.

EDIT: everything works well with extern

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits July 28, 2025 10:32
@picnixz
reduce exported scope
89c0e3c
@picnixz
Windows is not happy
a0fa849
@picnixz

picnixz commented Jul 28, 2025

Copy link
Copy Markdown
Member Author

Hum. Windows. Ok I don't know how to correct the project so using PyAPI_FUNC should be enough. I think we should find a better way to manage cryptographic modules on Windows as I feel it's confusing (I don't really know what to exactly do...)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Jul 28, 2025

Copy link
Copy Markdown
Member Author

!buildbot FIPS only

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Jul 28, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @picnixz for commit a0fa849 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136995%2Fmerge

The command will test the builders whose names match following regular expression: FIPS only

The builders matched are:

  • AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
  • AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz marked this pull request as ready for review July 28, 2025 09:17
@bedevere-app bedevere-app Bot added the awaiting core review label Jul 28, 2025
@picnixz
picnixz merged commit 45138d3 into python:main Jul 28, 2025
@picnixz
picnixz deleted the refactor/hashlib/extract-helpers-135341 branch July 28, 2025 09:28
@bedevere-app bedevere-app Bot removed the awaiting core review label Jul 28, 2025
@picnixz

picnixz commented Jul 28, 2025

Copy link
Copy Markdown
Member Author

I don't think I can backport this as I think it's an ABI change?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@freakboy3742

freakboy3742 commented Jul 31, 2025

Copy link
Copy Markdown
Contributor

Not sure why this hasn't flagged, but this seems to have caused a big problem for the Emscripten buildbot: https://buildbot.python.org/#/builders/1810/builds/234

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hoodmane

hoodmane commented Jul 31, 2025

Copy link
Copy Markdown
Contributor

Problem is we pass libhashlib.a to the linker twice which wasm-ld doesn't like.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba

zooba commented Jul 31, 2025

Copy link
Copy Markdown
Member

There shouldn't be any need for PyAPI_FUNC in internal headers - it's for public exported API (that needs to be used by apps loading python315.dll).

This seems like it should be a fully internal change, just refactoring sources?

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Aug 1, 2025

Copy link
Copy Markdown
Member Author

Oh so it did cause an issue. I actually tried using extern but the Windows build failed so I wasn't sure about it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Aug 1, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I will take care of this by making everything static inline just to prevent polluting main for a moment. Or I'll directly try to see how I can correctly use extern (I think the problem lies with the MSVC projects where I get duplicated symbols)

@zooba could you help me here setting up the projects?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Aug 1, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I tried something but I can only rely on the CI to test the build: #137301. I think the issue is that we have a hashlib project but we don't have HACL* projects. Is it sufficient for me to actually only specify the files to compile in the hashlib project (I don't know if it's possible to disable hashlib on Windows)? would they be picked up by the "parent" project correctly? EDIT: well apparently not. Let's discuss what I should do on the new PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added a commit to picnixz/cpython that referenced this pull request Aug 1, 2025
@picnixz
Revert "pythongh-131876: extract _hashlib helpers into a separate d…
4feb4d0
…irectory (python#136995)"

This reverts commit 45138d3.
gpshead pushed a commit that referenced this pull request Aug 1, 2025
@picnixz
gh-131876: Revert "gh-131876: extract _hashlib helpers into a separ…
fe0e921
…ate directory (#136995) (#137307)

Revert "gh-131876: extract `_hashlib` helpers into a separate directory (#136995)"

This reverts commit 45138d3.
@bedevere-bot

bedevere-bot commented Aug 1, 2025

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit fe0e921.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/1839) and take a look at the build logs.
  4. Check if the failure is related to this commit (fe0e921) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/1839

Failed tests:

  • test_free_threading

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 7, done.        
remote: Counting objects:  14% (1/7)        
remote: Counting objects:  28% (2/7)        
remote: Counting objects:  42% (3/7)        
remote: Counting objects:  57% (4/7)        
remote: Counting objects:  71% (5/7)        
remote: Counting objects:  85% (6/7)        
remote: Counting objects: 100% (7/7)        
remote: Counting objects: 100% (7/7), done.        
remote: Compressing objects:  14% (1/7)        
remote: Compressing objects:  28% (2/7)        
remote: Compressing objects:  42% (3/7)        
remote: Compressing objects:  57% (4/7)        
remote: Compressing objects:  71% (5/7)        
remote: Compressing objects:  85% (6/7)        
remote: Compressing objects: 100% (7/7)        
remote: Compressing objects: 100% (7/7), done.        
remote: Total 7 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'fe0e921817a7f96c62c91085884ab910859328ce'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at fe0e921817a gh-131876: Revert "gh-131876: extract `_hashlib` helpers into a separate directory (#136995) (#137307)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

make: *** [Makefile:2486: buildbottest] Error 2
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added a commit to picnixz/cpython that referenced this pull request Aug 2, 2025
@picnixz
pythongh-131876: extract _hashlib helpers into a separate directory (…
688ad54
…python#136995)

The `Modules/hashlib.h` helper file is now removed and split into multiple files:

* `Modules/_hashlib/hashlib_buffer.[ch]` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
  Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
picnixz added a commit to picnixz/cpython that referenced this pull request Aug 2, 2025
@picnixz
pythongh-131876: extract _hashlib helpers into a separate directory (…
a629e42
…python#136995)

The `Modules/hashlib.h` helper file is now removed and split into multiple files:

* `Modules/_hashlib/hashlib_buffer.[ch]` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
  Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
picnixz added a commit to picnixz/cpython that referenced this pull request Aug 2, 2025
@picnixz
pythongh-131876: extract _hashlib helpers into a separate directory (…
c7f7776
…python#136995)

The `Modules/hashlib.h` helper file is now removed and split into multiple files:

* `Modules/_hashlib/hashlib_buffer.[ch]` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
  Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
picnixz added a commit to picnixz/cpython that referenced this pull request Aug 2, 2025
@picnixz
pythongh-131876: extract _hashlib helpers into a separate directory (…
2fb8a6c
…python#136995)

The `Modules/hashlib.h` helper file is now removed and split into multiple files:

* `Modules/_hashlib/hashlib_buffer.h` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
  Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
@freakboy3742 freakboy3742 mentioned this pull request Aug 6, 2025
gh-128627: Use __builtin_wasm_test_function_pointer_signature for Emscripten trampoline #137470
Merged
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
@picnixz @Agent-Hellboy
pythongh-131876: extract _hashlib helpers into a separate directory (…
b707ead
…python#136995)

The `Modules/hashlib.h` helper file is now removed and split into multiple files:

* `Modules/_hashlib/hashlib_buffer.[ch]` -- Utilities for getting a buffer view and handling buffer inputs.
* `Modules/_hashlib/hashlib_fetch.h` -- Utilities used when fetching a message digest from a digest-like identifier.
  Currently, this file only contains common error messages as the fetching API is not yet implemented.
* `Modules/_hashlib/hashlib_mutex.h` -- Utilities for managing the lock on cryptographic hash objects.
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
@picnixz @Agent-Hellboy
pythongh-131876: Revert "pythongh-131876: extract _hashlib helpers …
ef4a1fc
…into a separate directory (python#136995) (python#137307)

Revert "pythongh-131876: extract `_hashlib` helpers into a separate directory (python#136995)"

This reverts commit 45138d3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@tiran tiran Awaiting requested review from tiran

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@picnixz @bedevere-bot @gpshead @freakboy3742 @hoodmane @zooba
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.