GeezerVison is showing github.com — gzip raising exception when closing with buffer backed by BytesIO · Issue #129726 · python/cpython
gzip raising exception when closing with buffer backed by BytesIO #129726DescriptionBug report
Bug description:
Hello,
the following snippet raises an exception in Python 3.13.2 while it's fine with Python 3.12.
import io
import gzip
def foo():
buffer = gzip.GzipFile(fileobj=io.BytesIO(), mode="w")
foo()Running this with python3.12 is silent, with Python 3.13.2 instead:
Exception ignored in: <gzip on 0x7fa4fd99c550>
Traceback (most recent call last):
File "/usr/lib/python3.13/gzip.py", line 359, in close
fileobj.write(self.compress.flush())
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
Activitychanged the title [-]gzip raising exception when closing empty buffer backed by bytesio[/-] [+]gzip raising exception when closing empty buffer backed by BytesIO[/+] on Feb 6, 2025changed the title [-]gzip raising exception when closing empty buffer backed by BytesIO[/-] [+]gzip raising exception when closing with buffer backed by BytesIO[/+] on Feb 7, 2025cmaloney commented on Feb 9, 2025 Explicit buffer.close() and del buffer in the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving to with gzip.GzipFile ... as buffer: also doesn't error. The issue seems to be that the io.BytesIO is cleaned up before the gzip.GzipFile.
The same behavior occurs with an explicit reference to the io.BytesIO:
def explicit_reference():
bio = io.BytesIO()
buffer = gzip.GzipFile(fileobj=bio, mode="w")Can work around it so no longer error at least by updating GzipFile.close() to check if self.fileobj.closed and fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order. xrmx commented on Feb 9, 2025 ContributorAuthor More actions
Explicit buffer.close() and del buffer in the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving to with gzip.GzipFile ... as buffer: also doesn't error.
Thanks for testing these, I cannot use the context manager but I may be able to close the buffer explictly.
The issue seems to be that the io.BytesIO is cleaned up before the gzip.GzipFile.
Yeah, that is my impression too
The same behavior occurs with an explicit reference to the io.BytesIO:
def explicit_reference():
bio = io.BytesIO()
buffer = gzip.GzipFile(fileobj=bio, mode="w")
Can work around it so no longer error at least by updating GzipFile.close() to check if self.fileobj.closed and fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order.
Yeah would be nice to keep the code working instead of not just crashing :) The gzip code should already have references to the fileobj so to me is surprising it gets deallocated while we are in in GzipFile.close. xrmx commented on Feb 10, 2025 ContributorAuthor More actions So my original code when getting the gzipped content is this so am not sure I can workaround this other than stop using GzipFile.
fileobj = buffer.fileobj # get a reference to the fileobj before closing the gzip file
buffer.close()
data = fileobj.getbuffer()
Irony of the comment 😅
The documentation explicitly says that calling GzipFile's close() does not close the fileobj, so I guess it also means it should not be freed:
Calling a GzipFile object’s close() method does not close fileobj, since you might wish to append more material after the compressed data.
This also allows you to pass an io.BytesIO object opened for writing as fileobj, and retrieve the resulting memory buffer using the io.BytesIO object’s getvalue() method.
cmaloney commented on Feb 10, 2025 xrmx commented on Feb 10, 2025 ContributorAuthor More actions cmaloney commented on Feb 10, 2025 Reacted by Riccardo Magliocchetti danifus commented on Feb 11, 2025 Reacted by Cody Maloney and Riccardo Magliocchetti xrmx commented on Feb 11, 2025 ContributorAuthor More actions Ok so with 3.12 and development mode I see the same warning, don't see the warning in 3.10 and 3.11 though:
$ python3.13 gzipfoo.py
Exception ignored in: <gzip on 0x7f39c7098580>
Traceback (most recent call last):
File "/usr/lib/python3.13/gzip.py", line 359, in close
fileobj.write(self.compress.flush())
ValueError: I/O operation on closed file.
$ python3.12 gzipfoo.py
$ python3.12 -X dev gzipfoo.py
Exception ignored in: <gzip on 0x7f1450669590>
Traceback (most recent call last):
File "/usr/lib/python3.12/gzip.py", line 357, in close
fileobj.write(self.compress.flush())
ValueError: I/O operation on closed file.
$ /usr/bin/python3.10 -X dev gzipfoo.py
$ /usr/bin/python3.11 -X dev gzipfoo.py
So there is something that changed the behavior in 3.12. xrmx commented on Feb 12, 2025 ContributorAuthor More actions @danifus thanks for your bisection. Any chance you can bisect with the -X dev param? I think there are good chances the code was failing already, just not printing the exception. vstinner commented on Feb 28, 2025 Fixed by change 7f39137. Backports to 3.12 and 3.13 will follow. xrmx commented on Feb 28, 2025 ContributorAuthor More actions vstinner commented on Feb 28, 2025 Maybe GzipFile should emit a ResourceWarning if the file is not closed explicitly to make sure that data are written on disk. cmaloney commented on Feb 28, 2025 I implemented a ResourceWarning on BufferedWriter checking for unwritten data, and just the CPython I/O tests it fires a lot. Even with conditions (only if raw isn't closed, etc) many cases to work through. I think worthwhile to help track down and cleanup inconsistencies at least in CPython itself. Not sure if it's going to be too noisy in general Python ecosystem codebases though. vstinner commented on Mar 2, 2025 The test suite can be fixed. The risk of losing data justify emitting ResourceWarning. josecsotomorales commented on Apr 9, 2025
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|