Description

bdarnell

opened on Jun 15, 2023

Issue body actions

Bug report

The change to add buffering to GzipFile.write (gh-89550, #101251) broke the GzipFile.flush method. The flush method previously called self.compress.flush, but now it only flushes the IO objects and not the compressor (as a side effect, the zlib_mode argument is now ignored, although in my case I only use the default). Flushing the compressor is necessary to create synchronization points that can be used to decompress part of the stream.

Here is a test script, reduced from Tornado's use of GzipFile (see tornadoweb/tornado#3278 for the way this manifests in Tornado's test suite):

import io
import gzip
import zlib

# Write two chunks to the same compressed stream. In real usage
# I send these chunks as two separate network messages, but in this
# test I just save them to two local variables.
data = io.BytesIO()
gzip_file = gzip.GzipFile(fileobj=data, mode="wb")
gzip_file.write(b"Hello World")
gzip_file.flush()
message1 = data.getvalue()
data.truncate(0)
data.seek(0)
gzip_file.write(b"Goodbye World")
gzip_file.close()
message2 = data.getvalue()

# Decode the two messages. Each one should decode separately,
# but in Python 3.12b2 the compressor was not flushed with
# Z_SYNC_FLUSH so the second message produces no output on its own
# and both messages are emitted when the second message is added
# to the decompressor's input.
#
# This results in the error
#   AssertionError: [b'', b'Hello WorldGoodbye World']
decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS)
messages = [decompressor.decompress(message1), decompressor.decompress(message2)]
assert messages == [b"Hello World", b"Goodbye World"], messages

Your environment

• CPython versions tested on: The bug is present in 3.12b2; the above script passes on 3.11 and earlier
• Operating system and architecture: macOS and Linux

Linked PRs

• GH-105808: Fix a regression introduced in GH-101251 #105910
• [3.12] GH-105808: Fix a regression introduced in GH-101251 (GH-105910) #105920

Activity

1. bdarnell

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jun 15, 2023
2. bdarnell

   added a commit that references this issue on Jun 15, 2023

   ci: Disable Python 3.12 in CI

   7640b8c
3. encukou

   added

   release-blocker

   on Jun 15, 2023
4. ezio-melotti

   added this to Release and Deferred blockers 🚫on Jun 15, 2023
5. github-project-automation

   moved this to Todo in Release and Deferred blockers 🚫on Jun 15, 2023
6. Yhg1s

   added a commit that references this issue on Jun 19, 2023

   pythonGH-105808: Fix a regression introduced in pythonGH-101251, causing

   b7ecf06
7. Yhg1s commented on Jun 19, 2023

   

   Yhg1s

   on Jun 19, 2023

   Member

   More actions

   Thanks! #105910 should fix the issue.
8. gpshead

   added

   3.12only security fixesonly security fixes

   on Jun 19, 2023
9. gpshead

   added a commit that references this issue on Jun 19, 2023

   GH-105808: Fix a regression introduced in GH-101251 (#105910)

   1858db7
10. miss-islington

    added a commit that references this issue on Jun 19, 2023

    pythonGH-105808: Fix a regression introduced in pythonGH-101251 (pyth…

    15b4976
11. Yhg1s

    added a commit that references this issue on Jun 19, 2023

    [3.12] GH-105808: Fix a regression introduced in GH-101251 (GH-105910) (

    225cc4c
12. gpshead commented on Jun 19, 2023

    

    gpshead

    on Jun 19, 2023

    Member

    More actions

    thanks for the report and PR!
13. gpshead

    closed this as completedon Jun 19, 2023
14. github-project-automation

    moved this from Todo to Done in Release and Deferred blockers 🚫on Jun 19, 2023
15. bdarnell

    added a commit that references this issue on Jun 22, 2023

    ci: Re-enable python 3.12

    acde88d
16. bdarnell commented on Jun 22, 2023

    

    bdarnell

    on Jun 22, 2023

    ContributorAuthor

    More actions

    Thanks for the fix! I can confirm that things are working again with 3.12 beta 3: tornadoweb/tornado#3288
17. cmaloney

    added a commit that references this issue on Mar 6, 2025

    pythongh-130806: emit ResourceWarning if GzipFile unclosed

    8bf1895
18. vstinner

    added a commit that references this issue on Mar 13, 2025

    gh-130806: Emit ResourceWarning if GzipFile unclosed (#130905)

    93089c0
19. plashchynski

    added a commit that references this issue on Mar 17, 2025

    pythongh-130806: Emit ResourceWarning if GzipFile unclosed (python#13…

    0158b99

Issue actions