GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-127298: Refactor test_hashlib for better usedforsecurity & openssl fips mode env support. by gpshead · Pull Request #127492 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Dec 1, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Ensure builtin hashlib implementations honor usedforsecurity=True when _hashlib is in FIPS mode #127298

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@gpshead
Refactor test_hashlib for better usedforsecurity & openssl fips mode …
59d9a85
…environment support.
@gpshead gpshead added tests Tests in the Lib/test dir skip news labels Dec 1, 2024
@gpshead

gpshead commented Dec 1, 2024

Copy link
Copy Markdown
Member Author

!buildbot FIPS

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Dec 1, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 59d9a85 🤖

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

  • AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
  • AMD64 RHEL8 FIPS No Builtin Hashes PR
  • AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
  • AMD64 CentOS9 FIPS No Builtin Hashes PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead
More hashlib robustness in strange "FIPS mode" environments.
bd46651
@gpshead

gpshead commented Dec 1, 2024

Copy link
Copy Markdown
Member Author

!buildbot FIPS

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Dec 1, 2024

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit bd46651 🤖

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

  • AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
  • AMD64 RHEL8 FIPS No Builtin Hashes PR
  • AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
  • AMD64 CentOS9 FIPS No Builtin Hashes PR
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead
gpshead force-pushed the hashlib-fips-oddmode branch from 7a5ab6c to bd46651 Compare December 2, 2024 03:42
@gpshead gpshead mentioned this pull request Dec 2, 2024
gh-84559: multiprocessing: detect if forkserver cannot work due to missing hmac-sha256 #127467
Closed
2 tasks
@gpshead
remove some obsolete comments
29a2012
@gpshead gpshead mentioned this pull request Dec 2, 2024
gh-127298: When in FIPS mode ensure builtin hashes check for usedforsecurity=False #127301
Open
@gpshead
remove debug print
e2bf70c
@gpshead gpshead changed the title Refactor test_hashlib for better usedforsecurity & openssl fips mode env support. gh-127298: Refactor test_hashlib for better usedforsecurity & openssl fips mode env support. Dec 2, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Dec 2, 2024
Ensure builtin hashlib implementations honor usedforsecurity=True when _hashlib is in FIPS mode #127298
Open
@gpshead gpshead self-assigned this Dec 2, 2024
@gpshead gpshead added stdlib Standard Library Python modules in the Lib/ directory and removed skip news labels Dec 2, 2024
@gpshead

gpshead commented Dec 2, 2024

Copy link
Copy Markdown
Member Author

#127467 is follow-on work to this that combined gets the FIPS mode buildbots passing in main.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead
gpshead marked this pull request as ready for review December 2, 2024 04:38
@gpshead
gpshead requested a review from tiran as a code owner December 2, 2024 04:38
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 2, 2024
@gpshead
gpshead requested review from vstinner and removed request for tiran December 2, 2024 04:38
nineteendo
nineteendo reviewed Dec 2, 2024
View reviewed changes
Comment thread Doc/library/hashlib.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@gpshead @nineteendo
docs typo
3f69411
Co-authored-by: Nice Zombies <[email protected]>
vstinner
vstinner reviewed Dec 2, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @stratakis @encukou

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/hashlib.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Library/2024-12-02-04-08-22.gh-issue-127298.8cpkfk.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

gpshead and others added 3 commits December 2, 2024 08:31
@gpshead @vstinner
remove unnecessary as in except statement,
2a5b7e8
Co-authored-by: Victor Stinner <[email protected]>
@gpshead
More details in NEWS
a6a75d6
@gpshead
news typo
3f9ec82
@stratakis

stratakis commented Dec 3, 2024

Copy link
Copy Markdown
Contributor

Thanks for the PR! I'll have a look at it and I can also provide ssh access to the FIPS buildbots if that would make things easier to debug.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Dec 5, 2024

Copy link
Copy Markdown
Member

I tested manually the change on RHEL8. 3 MD5 tests of test_hashlib are failing:

======================================================================
ERROR: test_case_md5_0 (test.test_hashlib.HashLibTestCase.test_case_md5_0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 547, in test_case_md5_0
    self.check(
    ~~~~~~~~~~^
        'md5', b'', 'd41d8cd98f00b204e9800998ecf8427e',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

======================================================================
ERROR: test_case_md5_1 (test.test_hashlib.HashLibTestCase.test_case_md5_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 553, in test_case_md5_1
    self.check(
    ~~~~~~~~~~^
        'md5', b'abc', '900150983cd24fb0d6963f7d28e17f72',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

======================================================================
ERROR: test_case_md5_2 (test.test_hashlib.HashLibTestCase.test_case_md5_2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 559, in test_case_md5_2
    self.check(
    ~~~~~~~~~~^
        'md5',
        ^^^^^^
    ...<2 lines>...
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

----------------------------------------------------------------------
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Dec 5, 2024

Copy link
Copy Markdown
Member

I built Python with ./configure --with-pydebug && make. MD5 is available:

[vstinner@python-builder-rhel8-fips cpython]$ ./python 
Python 3.14.0a2+ (heads/pr/127492:3f9ec82db, Dec  5 2024, 06:10:34) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5(usedforsecurity=False).hexdigest()
'd41d8cd98f00b204e9800998ecf8427e'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Dec 5, 2024

Copy link
Copy Markdown
Member

Suggested fix for file digest:

diff --git a/Lib/hashlib.py b/Lib/hashlib.py
index 44656c33a..93d602571 100644
--- a/Lib/hashlib.py
+++ b/Lib/hashlib.py
@@ -192,7 +192,7 @@ def __hash_new(name, data=b'', **kwargs):
     pass
 
 
-def file_digest(fileobj, digest, /, *, _bufsize=2**18):
+def file_digest(fileobj, digest, /, *, usedforsecurity=True, _bufsize=2**18):
     """Hash the contents of a file-like object. Returns a digest object.
 
     *fileobj* must be a file-like object opened for reading in binary mode.
@@ -206,9 +206,9 @@ def file_digest(fileobj, digest, /, *, _bufsize=2**18):
     # On Linux we could use AF_ALG sockets and sendfile() to archive zero-copy
     # hashing with hardware acceleration.
     if isinstance(digest, str):
-        digestobj = new(digest)
+        digestobj = new(digest, usedforsecurity=usedforsecurity)
     else:
-        digestobj = digest()
+        digestobj = digest(usedforsecurity=usedforsecurity)
 
     if hasattr(fileobj, "getbuffer"):
         # io.BytesIO object, use zero-copy buffer
diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index 1c1a0396c..a7cca0ba5 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -443,11 +443,13 @@ def check_file_digest(self, name, data, hexdigest):
             for digest in digests:
                 buf = io.BytesIO(data)
                 buf.seek(0)
-                self.assertEqual(
-                    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
-                )
+                digestobj = hashlib.file_digest(buf, digest,
+                                                usedforsecurity=False)
+                self.assertEqual(digestobj.hexdigest(), hexdigest)
+
                 with open(os_helper.TESTFN, "rb") as f:
-                    digestobj = hashlib.file_digest(f, digest)
+                    digestobj = hashlib.file_digest(f, digest,
+                                                    usedforsecurity=False)
                 self.assertEqual(digestobj.hexdigest(), hexdigest)
         finally:
             os.unlink(os_helper.TESTFN)
xnox reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xnox

xnox commented Dec 11, 2024

Copy link
Copy Markdown

@gpshead the suggestion from @vstinner looks reasonable in #127492 (comment) will you push that to this pr? Or should that change be done separately, with a separate blurb entry?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ghmj2417 ghmj2417 mentioned this pull request Dec 18, 2024
OpenSSL with FIPS enabled causes test_hashlib failures #128071
Closed
@xnox

xnox commented Apr 28, 2025

Copy link
Copy Markdown

@gpshead hi, my PRs had comments that you are taking over with these changes instead, but they now seem to have stalled. Are you still actively working on this and related PRs?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Apr 28, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Note that changing how we materialize a digest object should done carefully (see PEP-452). For new(), it should be fine as we're using our own hsahlib.new but for digest(), we should be careful here because not all digest constructors necessarily need to accept usedforsecurity (see the examples in https://docs.python.org/3/library/hashlib.html#hashlib.file_digest):

buf = io.BytesIO(b"somedata")
mac1 = hmac.HMAC(b"key", digestmod=hashlib.sha512)
digest = hashlib.file_digest(buf, lambda: mac1)
xnox reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@xnox

xnox commented Sep 8, 2025

Copy link
Copy Markdown

@gpshead Any updates on landing this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

github-actions Bot commented Apr 20, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 20, 2026
@gpshead

gpshead commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

I believe this PR is no longer relevant as is.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead closed this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
+1 more reviewer
@nineteendo nineteendo nineteendo left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

awaiting core review stale Stale PR or inactive for long period of time. stdlib Standard Library Python modules in the Lib/ directory tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@gpshead @bedevere-bot @stratakis @vstinner @xnox @picnixz @nineteendo
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.