GeezerVison

Adding skip news for now just for the CI to be green.

@msprotz The following code is not liked by MSVC:

  uint32_t l = 64U;
  KRML_CHECK_SIZE(sizeof (uint8_t), l);
  uint8_t key_block[l]; // <- this

I wonder whether you can do something (like inlining the variable instead) or tell me if it's possible to make MSVC happy.

I'm marking this PR as ready for review even though it fails to build on Windows. We'll address that later (for now, I want to get feedback on the build itself and how to get rid of .a files).

Ok, I need to think of a way to easily map hashlib<->openssl dispatch for HMAC. Converting into a draft for now.

Just to set expectations, @R1kM and I looked at what it would take to add streaming HMAC and we're relatively confident that it can be done within a few weeks. We have a couple more urgent things, then we'll get to it and post here as soon as we have an update, and then it'll simplify everything to have a single implementation that can do streaming too.

Ok, so I managed to fix the compilation and find a nice way to resolve OpenSSL names. However, since I need to bind OpenSSL for Windows builds, I'd like someone to create the project files for Windows since I don't have .NET installed (and honestly I don't want to install it; I could use a VM if I have time). So I'll just ignore the Windows build failures for now.

I looked at what it would take to add streaming HMAC and we're relatively confident that it can be done within a few weeks

No hurry, take your time! It'd be nice if the API for all HMAC-HASH functions are the same (or at least, that the functions can be stored using the same function pointer type).

It'd be nice if the API for all HMAC-HASH functions are the same (or at least, that the functions can be stored using the same function pointer type).

There are two styles of APIs, and we're not sure yet which one we want to go for. Your feedback is most welcome.

Agile APIs are of the form struct state { union { uint64_t *sha2_512_state; uint32_t *sha2_256 state; .... } hash_state; seen: uint64_t; hash_alg a; } void update(state *s, uint8_t *input, uint32_t input_len);

• a single update function for all choices of algorithms
• choice of algorithm is provided at state allocation-time (i.e. void malloc(hash_alg a);)
• extra level of indirection in the state due to the need to have a tagged union

Plain APIs are of the form typedef uint64_t *sha2_512 state; void update_512(sha2_512 *state, uint8_t *input, uint32_t input_len); and come in different variants (and different types) for all algorithms, i.e. there's also typedef uint32_t *sha2_256 state; void update_256(sha2_256 *state, uint8_t *input, uint32_t *input_len);, etc.

• one update function for each algorithm
• client manages the choice of algorithm (possibly keeping a tagged union themselves, or function pointers, whatever their style is)
• no extra level of indirection
• massive code duplication since each algorithm re-implements the state management logic

Thoughts on which one you'd prefer?

Thank you for the reply! Some questions:

massive code duplication since each algorithm re-implements the state management logic"

I assume however that performances are slightly better since you don't have indirection, right? How better do they fare in general? are plain APIs really better than agile ones?

Agile API seems good because I only need to specify the hash_alg value and be done with that. I'll need to think a bit more on that but I'm leaving for today and won't be available until tomorrow. I'll try to find a good example of how I plan to use it and then I'll come back to you!

I assume however that performances are slightly better since you don't have indirection, right? How better do they fare in general? are plain APIs really better than agile ones?

generally there's no noticeable performance impact unless you repeatedly hmac a lot of very small data, and repeatedly create/destroy the underlying python object, meaning you end up calling malloc/free a lot

the plain APIs exist for engineering reasons: you can package a single hmac algorithm along with its plain API, but if the agile API is the only API offered, a client like Python has to take all of the hmac algorithms in one go (since the toplevel agile API now contains references to every individual algorithm)

if you intend to overhaul all of the hmac algorithms in one go, then I would do agile API

FTR, I have a branch with streaming HMAC: https://github.com/picnixz/cpython/tree/hacl/HMAC-stream-99108. For now, I'm assuming that I have a plain API that can be used as follows:

/* Function pointer type for HACL* streaming HMAC state allocation */
typedef void *(*HACL_HMAC_state_malloc_func)(void);
/* Function pointer type for HACL* streaming HMAC state deallocation */
typedef void (*HACL_HMAC_state_free_func)(void *state);
/* Function pointer type for HACL* streaming HMAC state copy */
typedef void *(*HACL_HMAC_state_copy_func)(void *state);

/* Function pointer type for HACL* streaming HMAC setup functions. */
typedef hacl_exit_code
(*HACL_HMAC_setup_func)(void *state, uint8_t *key, uint32_t len);
/* Function pointer type for HACL* streaming HMAC update functions. */
typedef hacl_exit_code
(*HACL_HMAC_update_func)(void *state, uint8_t *buf, uint32_t len);
/* Function pointer type for HACL* streaming HMAC digest functions. */
typedef hacl_exit_code
(*HACL_HMAC_digest_func)(void *state, uint8_t *out);

• HACL_HMAC_state_malloc_func would allocate me the state for the HMAC-H algorithm
• HACL_HMAC_state_free_func would allow me to free it
• HACL_HMAC_state_copy_func would allow me to clone it
• HACL_HMAC_setup_func would allow me to feed the state with the user key (given at construction time). No message is given at that moment yet.
• HACL_HMAC_update_func would allow me to feed data in a streaming fashion.
• HACL_HMAC_digest_func would allow me to extract the digest from the state and putting it in out.

Ideally, I would like to expose the transformation function $f\colon\left\{0,1\right\}^*\longrightarrow\left\{0,1\right\}^B$ where $B$ is the block size so that users can also pass keys whose length exceeds $2^{32}-1$ or pad keys.

For now, the branch I mentioned assumes that the state consists of the message until now and the key (just to make it work).

Now, using an agile API, I could lighten the current interface to avoid storing those function pointers and only store the hash_alg. However, I'd like to be able to feed the initial key without having to feed the message, as well as the possibility to clone a state (since we have a .copy() method).

I'm coming back online after being off last week. Made some good progress on an agile HMAC interface, hopefully first prototype by the end of the week. It will be agile, meaning you won't have to deal with function pointers, the dynamic dispatch will be internal. The malloc function will take an algorithm and a key, and you will also have access to the copy function you mentioned. This should all greatly simplify your life.

Hoping to send a link to the PR soon.

The malloc function will take an algorithm and a key, and you will also have access to the copy function you mentioned. This should all greatly simplify your life.

That would be perfect! Thanks in advance!

I still need to know how I can make the Windows build work :( But I should have prepared everything to support the agile API.

Just for those subscribed to this pull request. I plan to re-open a fresh one using picnixz/cpython@feat/hmac/hacl-99108 in order to incorporate both streaming and 1-shot HMAC at the same time. However, we can continue the design discussion on this PR until the HACL* part is done.

Closing in favor of #130157.