GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-146311: Reject non-canonical padding bits in base32, 64, & 85 decoding by gpshead · Pull Request #146312 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Mar 22, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Summary

Add canonical=False keyword argument to a2b_base64, a2b_base32, a2b_base85, and a2b_ascii85 (and their base64 module wrappers). When canonical=True, non-canonical encodings are rejected per RFC 4648 section 3.5.

This is independent of strict_mode.

For base85/ascii85, the check also rejects single-character final groups (never produced by a conforming encoder) and verifies partial group padding matches what the encoder would produce.

  • Issue: binascii base64 & base32 decode ignore excess padding bits in their input #146311

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot mentioned this pull request Mar 22, 2026
binascii base64 & base32 decode ignore excess padding bits in their input #146311
Closed
gpshead and others added 2 commits March 22, 2026 15:05
@gpshead @claude
Add tests for rejecting non-zero padding bits in base64/base32
1a71757
RFC 4648 section 3.5 allows decoders to reject encoded data containing
non-zero pad bits. Both a2b_base64 (strict_mode=True) and a2b_base32
currently silently discard non-zero trailing bits instead of raising
binascii.Error. These tests document the expected behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Reject non-zero padding bits in base64/base32 decoding
0ca2563
Add leftchar validation after the main decode loop in a2b_base64
(strict_mode only) and a2b_base32 (always). Fix existing test data
that incidentally had non-zero padding bits to use characters with
zero trailing bits while preserving the same decoded output.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead
gpshead force-pushed the gh-146311-nonzero-padding-bits branch from 8451e22 to 0ca2563 Compare March 22, 2026 22:07
@gpshead @claude
Fix test_base64 test data with non-zero padding bits
615b227
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead

gpshead commented Mar 22, 2026

Copy link
Copy Markdown
Member Author

discussing if base32 needs strict_mode on the issue. not adding a NEWS entry until that is decided.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead gpshead self-assigned this Mar 22, 2026
gpshead and others added 3 commits April 4, 2026 22:40
@gpshead
Merge remote-tracking branch 'origin/main' into pythongh-146311-nonze…
1bf5c75
…ro-padding-bits
@gpshead @claude
Add canonical= kwarg to base64/base32/base85/ascii85 decoders
819e14e
Gate non-zero padding bits rejection behind a new canonical= keyword
argument independent of strict_mode, per discussion on pythongh-146311.

Per RFC 4648 section 3.5 ("Canonical Encoding"), decoders MAY reject
encodings where pad bits are not zero. The new canonical=True flag
enables this check for a2b_base64, a2b_base32, a2b_base85, and
a2b_ascii85.

For base85/ascii85, the canonical check also rejects single-character
final groups (never produced by a conforming encoder) and verifies
that partial group encodings match what the encoder would produce.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Add 'canonical' to global strings tables
4b7c6ae
The _Py_ID(canonical) identifier used by the clinic-generated
argument parsing code needs to be registered in the global strings.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead gpshead changed the title gh-146311: Reject non-zero padding bits in base64/base32 decoding gh-146311: Reject non-canonical padding bits in base32, 64, & 85 decoding Apr 4, 2026
gpshead and others added 4 commits April 4, 2026 23:51
@gpshead @claude
Remove incorrect RFC 4648 references from base85/ascii85
308433a
RFC 4648 only covers base16, base32, and base64. The canonical
encoding concept applies to base85 but is not defined by that RFC.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Simplify base85 canonical check using integer division
9b78bdd
Replace the re-encode-and-compare loops with a quotient comparison:
two divisions by 85**n_pad tell us whether the decoded uint32 and
the zero-padded output bytes share the same leading base-85 digits.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Improve base85/ascii85 canonical test coverage
101edf6
- Test non-canonical rejection for all partial group sizes (2/3/4 chars)
- Test digit-0 1-char group for ascii85 (exercises chunk_len==0 guard)
- Test boundary byte values (\x00, \xff) at each group size

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Add hypothesis tests for canonical encoding
b618655
- Round-trip tests: encoder always produces canonical output (base64,
  base32, base85, ascii85)
- Uniqueness tests: for base85/ascii85 partial groups, sweep all 85
  last-digit values and verify exactly one decodes to the original
  payload with canonical=True

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
serhiy-storchaka
serhiy-storchaka reviewed Apr 5, 2026
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM technically, but I have two notes:

  • The check for single-character final group should be unconditional, like in other codecs. This is the part of the specification, without MAY.
  • "Canonical" Ascii85/Base85 encoding is not defined, this is a projection. And in case of Ascii85 many other deviations of "canonical" encoding are accepted by default and not checked. As minimum, we should use "canonical" in quotes here, or don't use that word for these encodings.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Modules/binascii.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/binascii.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

gpshead and others added 4 commits April 5, 2026 18:48
@gpshead
Merge branch 'main' into pythongh-146311-nonzero-padding-bits
e8498dd
@gpshead @claude
Reject 1-char base85/ascii85 groups unconditionally
b5391bd
Per the PLRM spec (section 3.13.3), a final partial 5-tuple containing
only one character is an encoding violation. Move this check outside
the `canonical=` guard so it is always enforced.

Also change chunk_len and i from Py_ssize_t to int per review feedback.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Enforce z-abbreviation for all-zero groups in ascii85 canonical mode
2a1d91d
When canonical=True, reject '!!!!!' (five zero digits) in favor of
the 'z' abbreviation. The PLRM spec defines 'z' as the representation
for all-zero groups, so '!!!!!' is a non-canonical encoding.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@gpshead @claude
Update docs for base85/ascii85 canonical and 1-char group changes
dea4982
- Document that single-character final groups are always rejected
- Add versionchanged:: next markers for the behavioral change
- Update canonical= description for ascii85 to mention z-abbreviation
- Update canonical= description for base85

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
serhiy-storchaka
serhiy-storchaka reviewed Apr 6, 2026
View reviewed changes
Comment thread Doc/library/base64.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/binascii.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/binascii.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Apr 6, 2026

Copy link
Copy Markdown
Member

discussing if base32 needs strict_mode on the issue. not adding a NEWS entry until that is decided.

It was decided to use canonical, right?

Please update also What's New.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead added 3 commits April 25, 2026 11:48
@gpshead
Merge remote-tracking branch 'origin/main' into pythongh-146311-nonze…
ea1afef
…ro-padding-bits

# Conflicts:
#	Doc/library/base64.rst
@gpshead
Address PR review nits: doc ordering, partial-group wording, NEWS, wh…
00fb6fc
…atsnew

- Reorder versionchanged param lists alphabetically in base64.rst and
  binascii.rst (canonical, ignorechars, padded).
- Restore "two to five" in a2b_ascii85/a2b_base85 partial-group text;
  partial finals are 2-4 chars but a complete final group is 5.
- Drop redundant versionchanged blocks from a2b_ascii85/a2b_base85
  since both functions are new in 3.15.
- Add Misc/NEWS.d entry and Doc/whatsnew/3.15.rst bullets covering the
  new canonical kwarg and the unconditional 1-char rejection.
@gpshead
Define "canonical" locally for ascii85/base85 docs
1146806
The PLRM (Adobe Ascii85) and the various base85 alphabets do not define
a "canonical" form. Where the underlying spec doesn't ground the term,
the doc text now says explicitly that "canonical" here means "the
encoding the corresponding b2a_* function would produce."

Base16/32/64 entries are unchanged: RFC 4648 grounds the term there.
@gpshead

gpshead commented Apr 25, 2026

Copy link
Copy Markdown
Member Author

It was decided to use canonical, right?

yeah given our discussion on the issue that seemed like the best name. I also updated the *85 API docs to cover what "canonical" specifically means to us given that isn't defined by any of the looser specs for those.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead
gpshead marked this pull request as ready for review April 25, 2026 19:15
@gpshead
gpshead requested a review from AA-Turner as a code owner April 25, 2026 19:15
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 25, 2026
@gpshead
gpshead requested a review from serhiy-storchaka April 25, 2026 19:15
serhiy-storchaka
serhiy-storchaka approved these changes Apr 25, 2026
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Apr 25, 2026
@gpshead
gpshead merged commit c5fcdb4 into python:main Apr 25, 2026
60 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Apr 25, 2026
@Viicos Viicos mentioned this pull request Aug 4, 2026
Incorrect 3.15 what's new entry for base64 module #155193
Closed
@advitrocks9 advitrocks9 mentioned this pull request Aug 23, 2026
gh-155193: Fix the *canonical* decoder lists in the 3.15 What's New and NEWS #156302
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@gpshead @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.