GeezerVison

All commit authors signed the Contributor License Agreement.

It's a year later, and Z85 support has been added to base64 in the meantime. So while bringing this PR up to date with main, I added Z85 support to it as well.

For reference, this is the benchmark run that led me to do so.

# After merging main but before adding Z85 support to this PR
(cpython-b85) $ python bench_b85.py 64
b64encode : 67108864 b in 0.121 s (527.435 MB/s) using 42.667 MB
b64decode : 89478488 b in 0.309 s (276.188 MB/s) using 56.889 MB
a85encode : 67108864 b in 0.297 s (215.150 MB/s) using 0.000 MB
a85decode : 83886080 b in 0.205 s (390.751 MB/s) using 0.000 MB
b85encode : 67108864 b in 0.106 s (604.359 MB/s) using 0.000 MB
b85decode : 83886080 b in 0.204 s (393.040 MB/s) using 0.000 MB
z85encode : 67108864 b in 0.204 s (313.610 MB/s) using 80.000 MB
z85decode : 83886080 b in 0.300 s (266.670 MB/s) using 100.000 MB

The existing Z85 implementation translates from the standard base85 alphabet to Z85 after the fact and within Python, so it was already benefiting from this PR but with substantial performance and memory usage overhead. That overhead is now gone.

All commit authors signed the Contributor License Agreement.

PR has been rebased onto main at 78cfee6 with squashing.

Note that attempting to decode Ascii85, base85, or Z85 data of length 1 mod 5 now produces an error, as no encoder would emit such data. This should be the only significant externally visible difference compared to the old implementations.

I believe you have to document this change.

Note that attempting to decode Ascii85, base85, or Z85 data of length 1 mod 5 now produces an error, as no encoder would emit such data. This should be the only significant externally visible difference compared to the old implementations.

I believe you have to document this change.

Fair point, I could do that.

In case anyone argues for keeping the old behavior (silently ignoring length 1 mod 5), I won't do it just yet.

The PR has been updated to preserve the existing base 85 Python functions in base64 and modify the new base 85 C functions in binascii to closely match their behavior. Notably, trying to decode data of length 1 mod 5 is no longer an error.

The PR has been updated to address most reviewer comments. I even found some minor decoding performance gains. A bit more polish and this one will be done!

@serhiy-storchaka, sorry I was busy and didn't get to your feedback this past week. Looks good.

One last thing from me - I noticed you changed the clinic input for the ignorechars param in binascii.a2b_base64() to ignorechars: Py_buffer(py_default="<unrepresentable>") = None. I found a discussion thread about <unrepresentable> being for C arguments that are NULL, but it looks like the way you have it, ignorechars actually isn't NULL on the C side in binascii.a2b_base64(), so I don't know what's going on and if this change should be carried over to binascii.a2b_ascii85() or not.

No problem, I pushed my changes because there are several other issues wait for this change (Base32 codec, wrapcol and ignorechars for more codecs, maybe other options). Your work is great, sorry that it took several years to review it.

binascii.a2b_base64() is special case. It is non-strict by default, but this is a vulnerability. Since ignorechars is a new parameter, we can switch to strict mode by default if it is used. So, showing ignorechars=b'') in the signature would be a lie, because passing ignorechars=b'' and not passing the ignorechars argument have different semantic. So we use a special placeholder. This is temporary, when we made binascii.a2b_base64() strict by default, we could just make b'' the default value for ignorechars. The Ascii85 codec does not have such legacy.

There is also a bug in the Py_buffer converter. I only discovered it when worked on these issues. It only accepts None as the default value, even if None is not valid value for Py_buffer. It should accept NULL and a bytes literal. Then these declarations would look more natural.

Thanks for the explanation about clinic input. And thanks again for accepting my PR and making the process pleasant. The timeline is indeed unfortunate but also isn't anyone's fault.

@kangtastic, are you also interested in implementing the Base32 codec in C? This will not only improve its performance and memory usage, but also make it easier to add more features in the future.

I'm going to do it eventually, but I'm short on time right now. If you're interested, I promise a quick review.

@serhiy-storchaka, I'm also short on time for the next ~2 weeks, but I might be interested after that. I'll ping you in this issue when I know for sure.

@serhiy-storchaka, I'm interested in writing that Base32 implementation after all. Let me know if it's still needed/if you're looking for anything specific.

It would be great!

I asked you because I see that you are quite capable of this, and it might also be interesting for you.

As for anything specific, I suggest to keep the code for casefold and map01 in Python for now, like in the Base16 codec.

@serhiy-storchaka, OK, I'll get started and do it as you suggest.

See also #145980. It would be enough to only implement a single pair of functions for Base32.

@serhiy-storchaka, did you want me to wait for #145981 to be finalized and merged before I create an issue and a PR for the base32 accelerator? It's almost finished except for testing, etc.

You can open an issue and start working on a PR, then rebase your branch after merging #145981. Just leave the code for base32hex for now.

I am also working on a PR which depends on #145981 and on your future code.