Description

assignee = None
closed_at = None
created_at = <Date 2021-10-06.08:42:34.497>
labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library', '3.11']
title = 'GzipFile.write should be buffered'
updated_at = <Date 2021-10-06.08:59:49.938>
user = 'https://github.com/rhpvorderman'

activity = <Date 2021-10-06.08:59:49.938>
actor = 'marcelm'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-06.08:42:34.497>
creator = 'rhpvorderman'
dependencies = []
files = []
hgrepos = []
issue_num = 45387
keywords = []
message_count = 1.0
messages = ['403289']
nosy_count = 2.0
nosy_names = ['marcelm', 'rhpvorderman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue45387'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

1. rhpvorderman commented on Oct 6, 2021

   

   rhpvordermanmannequin

   on Oct 6, 2021

   MannequinAuthor

   More actions

   Please consider the following code snippet:

       import gzip
       import sys
   
       with gzip.open(sys.argv[1], "rt") as in_file_h:
           with gzip.open(sys.argv[2], "wt", compresslevel=1) as out_file_h:
               for line in in_file_h:
                   # Do processing on line here
                   modified_line = line
                   # End processing
                   out_file_h.write(modified_line)

   This is very slow, due to write being called for every line. This is the current implementation of write:
   

   cpython/Lib/gzip.py

   Line 272 in c379bc5

   def write(self,data):

   It:

   • Checks if the file is not closed
   • Checks if the correct mode is set
   • Checks if the file is not closed (again, but in a different way)
   • Checks if the data is bytes, bytearray or something that supports the buffer protocol
   • Gets the length
   • Compresses the data
   • updates the size and offset
   • updates the checksum

   Doing this for every line written is very costly and creates a lot of overhead in Python calls. We spent a lot of time in Python and a lot less in the fast C zlib code that does the actual compression.

   This problem is already solved on the read side. A _GzipReader object is used for reading. This is put in an io.BufferedReader which is used as the underlying buffer for GzipFile.read. This way, lines are read quite fast from a GzipFile without the checksum etc. being updated on every line read.

   A similar solution should be written for write.
   I volunteer (I have done some other work on gzip.py already), although I cannot give an ETA at this time.
2. rhpvorderman

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   3.7 (EOL)end of lifeend of life

   3.8 (EOL)end of lifeend of life

   3.9 (EOL)end of lifeend of life

   3.10 (EOL)end of lifeend of life

   3.11only security fixesonly security fixes

   performancePerformance or resource usagePerformance or resource usage

   on Oct 6, 2021
3. 9 remaining items
4. gpshead

   added a commit that references this issue on May 8, 2023

   gh-89550: Buffer GzipFile.write to reduce execution time by ~15% (#10…

   9af4854
5. jbower-fb

   added a commit that references this issue on May 8, 2023

   pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (

   a49c8a8
6. carljm

   added 4 commits that reference this issue on May 9, 2023

   Merge branch 'main' into inlinecomp2

   76077cd

   Merge branch 'main' into fixsendfamily

   a614216

   Merge branch 'main' into supermon

   c48eee3

   Merge branch 'main' into fixloadsuperspec

   cf11908
7. hugovk commented on Dec 11, 2023

   

   hugovk

   on Dec 11, 2023

   Member

   More actions

   A PR has been merged, thanks for the report!
8. hugovk

   closed this as completedon Dec 11, 2023
9. github-project-automation

   moved this to Done in Compression issues 🗜on Dec 11, 2023
10. cmaloney

    added a commit that references this issue on Feb 12, 2025

    pythongh-129726: Break `gzip.GzipFile` reference loop

    2d67d94
11. vstinner

    added a commit that references this issue on Feb 28, 2025

    gh-129726: Break `gzip.GzipFile` reference loop (#130055)

    7f39137
12. miss-islington

    added 2 commits that reference this issue on Feb 28, 2025

    pythongh-129726: Break `gzip.GzipFile` reference loop (pythonGH-130055)

    7178c06

    pythongh-129726: Break `gzip.GzipFile` reference loop (pythonGH-130055)

    ec34427
13. vstinner

    added 2 commits that reference this issue on Feb 28, 2025

    [3.12] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

    500ea3b

    [3.13] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

    ad97027

Issue actions