Description

assignee = None
closed_at = None
created_at = <Date 2018-02-26.21:08:51.426>
labels = ['type-bug', 'library']
title = 'zipimport fails when the ZIP archive contains more than 65535 files'
updated_at = <Date 2018-09-19.07:19:25.103>
user = 'https://bugs.python.org/mchouza'

activity = <Date 2018-09-19.07:19:25.103>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-02-26.21:08:51.426>
creator = 'mchouza'
dependencies = []
files = []
hgrepos = []
issue_num = 32959
keywords = []
message_count = 3.0
messages = ['312957', '313044', '325718']
nosy_count = 4.0
nosy_names = ['twouters', 'mchouza', 'serhiy.storchaka', 'bbayles']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32959'
versions = ['Python 3.6']

Activity

1. mchouza commented on Feb 26, 2018

   

   mchouzamannequin

   on Feb 26, 2018

   MannequinAuthor

   More actions

   When trying to import a module from a ZIP archive containing more than 65535 files, the import process fails:

   $ python3 -VV
   Python 3.6.4 (default, Jan  6 2018, 11:49:38) 
   [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
   $ cat create_zips.py 
   from zipfile import ZipFile
   with ZipFile('below.zip', 'w') as zfp:
       for i in range(65535):
           zfp.writestr('m%d.py' % i, '')
   with ZipFile('over.zip', 'w') as zfp:
       for i in range(65536):
           zfp.writestr('m%d.py' % i, '')
   $ python3 create_zips.py 
   $ python -m zipfile -l below.zip | (head -2 && tail -2)
   File Name                                             Modified             Size
   m0.py                                          2018-02-26 20:57:32            0
   m65533.py                                      2018-02-26 20:57:36            0
   m65534.py                                      2018-02-26 20:57:36            0
   $ python -m zipfile -l over.zip | (head -2 && tail -2)
   File Name                                             Modified             Size
   m0.py                                          2018-02-26 20:57:36            0
   m65534.py                                      2018-02-26 20:57:40            0
   m65535.py                                      2018-02-26 20:57:40            0
   $ PYTHONPATH=below.zip python3 -c 'import m0'
   $ PYTHONPATH=over.zip python3 -c 'import m0'
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   ModuleNotFoundError: No module named 'm0'

   I think the problem is related to the zipimport module not handling the 'zip64 end of central directory record'.
2. mchouza

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Feb 26, 2018
3. Yhg1s commented on Feb 28, 2018

   

   Yhg1s

   on Feb 28, 2018

   Member

   More actions

   FWIW, yes, this is because zipimport doesn't support ZIP64, and doesn't even flag it as an error when the ZIP requires it. Instead it skips files; the ZIP64 format works by setting the fields that would overflow to the maximum value as a signal that the real value should be read from the ZIP64 records, which means it's still a valid non-64-bit ZIP file, just a truncated one.

   Adding ZIP64 support to zipimport is not trivial, and requires breaking some behaviour that's bad (not according to the ZIP spec) but actually covered by zipimport's tests: prepending arbitrary data to the ZIP file by just concatenating them together (rather than rewriting the ZIP archive's offsets). I don't think there's going to be an easy fix here, although I am working on open-sourcing a replacement zipimport that I wrote for work, which does support ZIP64 properly (and also fixes some other fundamental flaws in zipimport). I hope to have that done soon, and we can discuss replacing zipimport with that module in Python 3.8 or later.
4. serhiy-storchaka commented on Sep 19, 2018

   

   serhiy-storchaka

   on Sep 19, 2018

   Member

   More actions

   zipimport have been rewritten in pure Python (bpo-25711).
5. ezio-melotti

   transferred this issue fromon Apr 10, 2022
6. gpshead

   added a commit that references this issue on Mar 28, 2024

   gh-89739: gh-77140: Support zip64 in zipimport (GH-94146)

   b448982
7. gpshead

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Mar 28, 2024
8. gpshead commented on Mar 28, 2024

   

   gpshead

   on Mar 28, 2024

   Member

   More actions

   This feature should appear in the next 3.13 pre-release.
9. gpshead

   closed this as completedon Mar 28, 2024
10. NGRsoftlab

    added 2 commits that reference this issue on Apr 2, 2024

    Squashed commit of the following:

    1b5fe4c

    Squashed commit of the following:

    a016a54
11. diegorusso

    added a commit that references this issue on Apr 17, 2024

    pythongh-89739: pythongh-77140: Support zip64 in zipimport (pythonGH-…

    3c8fa9d

Issue actions