GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-132983: Don't allow trailer data in ZstdFile by Rogdham · Pull Request #133736 · python/cpython

Conversation

@Rogdham

@Rogdham Rogdham commented May 9, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

We previously made sure that an exception is raised when decompressing trailer data with decompress:

>>> from compression.zstd import compress, decompress
>>> invalid = compress(b'xxx') + b'yyy'
>>> decompress(invalid)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    decompress(invalid)
    ~~~~~~~~~~^^^^^^^^^
  File "/redacted/Lib/compression/zstd/__init__.py", line 157, in decompress
    results.append(decomp.decompress(data))
                   ~~~~~~~~~~~~~~~~~^^^^^^
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor

Indeed, the Zstandard specification says “Zstandard compressed data is made of one or more frames”, and it does not say that random data can be added at the end.

However, this is not the case in ZstdFile / zstd.open:

>>> from compression.zstd import ZstdFile
>>> from io import BytesIO
>>> ZstdFile(BytesIO(invalid)).read()
b'xxx'

After this PR, the last call becomes:

>>> ZstdFile(BytesIO(invalid)).read()
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    ZstdFile(BytesIO(invalid)).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/Lib/compression/zstd/_zstdfile.py", line 176, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 91, in read
    data = self._decompressor.decompress(rawblock, size)
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor
  • Issue: Implement PEP 784 - Adding Zstandard to the Python standard library #132983

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@Rogdham
ZstdFile: don't allow trailer data
98d2b86
@bedevere-app bedevere-app Bot mentioned this pull request May 9, 2025
Implement PEP 784 - Adding Zstandard to the Python standard library #132983
Closed
13 tasks done
@Rogdham
Rogdham marked this pull request as ready for review May 9, 2025 09:36
@bedevere-app bedevere-app Bot added the awaiting review label May 9, 2025
@AA-Turner AA-Turner added the needs backport to 3.14 bugs and security fixes label May 9, 2025
@emmatyping

emmatyping commented May 9, 2025

Copy link
Copy Markdown
Member

The current behavior matches LZMA. I think unlike decompress which is handed what is necessarily a zstd stream of one or more frames, with ZstdFile, a user may be parsing a format which has additional information after a zstd stream.

>>> from lzma import LZMAFile, compress
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid)).read()
b'foo'
>>>
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@Rogdham

Rogdham commented May 9, 2025

Copy link
Copy Markdown
Contributor Author

You are right this is the case for LZMAFile with format FORMAT_AUTO (which is the default) and also for BZ2File.

However, LZMAFile with format FORMAT_XZ as well as GzipFile raise an exception in that case.

>>> from lzma import LZMAFile, compress, FORMAT_XZ
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/lzma.py", line 208, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/_compression.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
                   "end-of-stream marker was reached")
EOFError: Compressed file ended before the end-of-stream marker was reached
emmatyping reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
emmatyping
emmatyping approved these changes May 9, 2025
View reviewed changes

@emmatyping emmatyping left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay this looks good then!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels May 9, 2025
@emmatyping emmatyping added the skip news label May 9, 2025
@Rogdham

Rogdham commented May 9, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

In addition, consider decompress(compress(b"xxx") + b"yyy"):

  • returns b"xxx" on: lzma (format FORMAT_AUTO), bz2
  • raises an exception on: lzma (format FORMAT_XZ), gzip

Since for zstd we raise an exception on that, I would say to do the same for ZstdFile to be consistent.
emmatyping reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AA-Turner
AA-Turner merged commit 50b5370 into python:main May 10, 2025
@miss-islington-app

miss-islington-app Bot commented May 10, 2025

Copy link
Copy Markdown

Thanks @Rogdham for the PR, and @AA-Turner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting core review label May 10, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 10, 2025
@Rogdham @miss-islington
pythongh-132983: Don't allow trailer data in ZstdFile (pythonGH-133736)
0c55b02 
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <[email protected]>
@bedevere-app

bedevere-app Bot commented May 10, 2025

Copy link
Copy Markdown

GH-133799 is a backport of this pull request to the 3.14 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label May 10, 2025
AA-Turner pushed a commit that referenced this pull request May 10, 2025
@miss-islington @Rogdham
[3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) (#…
99ca086 
…133799)

gh-132983: Don't allow trailer data in ZstdFile (GH-133736)
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <[email protected]>
@Rogdham
Rogdham deleted the zstdfile-trailer-exception branch May 10, 2025 06:45
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@Rogdham @Pranjal095
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
3882f69
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@Rogdham @taegyunkim
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
11bf99d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emmatyping emmatyping emmatyping approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@Rogdham @emmatyping @AA-Turner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.