GeezerVison is showing github.com — Add a file_digest() function in hashlib · Issue #89313 · python/cpython
Add a file_digest() function in hashlib #89313Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-09-09.10:03:17.355>
labels = ['type-feature', 'library', '3.11']
title = 'Add a file_digest() function in hashlib'
updated_at = <Date 2022-03-22.15:41:04.324>
user = 'https://github.com/tarekziade' bugs.python.org fields:
activity = <Date 2022-03-22.15:41:04.324>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-09-09.10:03:17.355>
creator = 'tarek'
dependencies = []
files = []
hgrepos = []
issue_num = 45150
keywords = ['patch']
message_count = 14.0
messages = ['401457', '401461', '401462', '415138', '415139', '415141', '415320', '415321', '415324', '415326', '415336', '415356', '415754', '415793']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'tarek', 'python-dev', 'Aur.Saraf', 'miss-islington']
pr_nums = ['28252', '31928', '31930', '32046']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45150'
versions = ['Python 3.11'] Activitytarekziade commented on Sep 9, 2021 MannequinAuthor More actions I am proposing the addition of a very simple helper to return the hash of a file. tiran commented on Sep 9, 2021 Hey Tarek, long time no see!
-
the _sha256 module is optional, can be disabled and is not available in some distributions.
-
I also don't like to use sha256 has default. It's slow, even slower than sha512. Any default makes it also harder to upgrade to a better, more secure default in the future.
-
like hmac.new() a file_digest() should accept PEP-452-compatible arguments and hash name as digstmod argument, not just a callable.
-
a filename argument prevents users from passing in file-like objects like BytesIO.
-
4096 bytes chunk size is very conservative. The call overhead for read() and update() may dominate the performance of the function.
-
The hex argument feels weird.
In a perfect world, the hash and hmac objects should get an "update_file" method. The OpenSSL-based hashes could even release the GIL and utilize OpenSSL's BIO layer to avoid any Python overhead. tarekziade commented on Sep 9, 2021 MannequinAuthor More actions Hey Christian, I hope things are well for you!
Thanks for all the precious feedback, I'll rework the patch accordingly SonOfLilit commented on Mar 14, 2022 Tarek,
Are you still working on this? Would you like me to take over?
Aur tarekziade commented on Mar 14, 2022 MannequinAuthor More actions @aur, go for it, I started to implement it and got lost into the details for each backend.. SonOfLilit commented on Mar 14, 2022 SonOfLilit commented on Mar 16, 2022 PR contains a draft implementation, would appreciate some review before I implement the same interface on all builtin hashes as well as OpenSSL hashes. SonOfLilit commented on Mar 16, 2022 I don't think HMAC of a file is a common enough use case to support, but I have absolutely no problem conceding this point, the cost of supporting it is very low.
I/O in C is a world of pain in general. In the specific case of io.RawIOBase objects (non-buffered binary files) to my understanding it's not that terrible (am I right? Does my I/O code work as-is?). To my understanding, providing a fast path just for this case that calculates the hash without taking the GIL for every chunk would be very nice to have for many use cases.
Now, we could just be happy with file_digest() having an if for isinstance(io.RawIOBase) that chooses a fast code path silently. But since non-buffered binary files are so hard to tell apart from other types of file-like objects, as a user of this code I would like to have a way to say "I want the fast path, please raise if I accidentally passed the wrong things and got the regular path". We could have file_digest('sha256', open(path, 'rb', buffered=0), ensure_fast_io=True), but I think for this use case raw_file_digest('sha256', open(path, 'rb', buffered=0)) is cleaner.
In all other cases you just call file_digest(), probably get the Python I/O and not the C I/O, and are still happy to have that loop written for you by someone who knows what they're doing.
For the same reason I think the fast path should only support hash names and not constructors/functions/etc', which would complicate it because new-object-can-be-accessed-without-GIL wouldn't necessarily apply.
Does this make sense? miss-islington commented on Mar 22, 2022 tiran commented on Mar 22, 2022 pablogsal commented on Aug 13, 2022 @tiran Can you made a PR adding the file_digest to the 3.11 what's new, please? calestyo commented on Jun 23, 2023 Hey folks.
I know this is closed and perhaps I should simply file a new request... but would you consider to extend the interface of that function to (efficiently) calculate a file's hashsum for multiple algorithms (i.e. without reading it once for every algo)?
One could perhaps do so by accepting some array for digest and return a dict where the alogo name is the key and the hashvalue the value.
Or perhaps something smarter ^^
Cheers,
Chris. gpshead commented on Jun 23, 2023 Please file a new feature request issue here for that.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|