GeezerVison is showing github.com — Rewrite zipimport from scratch · Issue #69897 · python/cpython
Rewrite zipimport from scratch #69897Description
| BPO |
25711 |
| Nosy |
@Yhg1s, @warsaw, @brettcannon, @gpshead, @ncoghlan, @pmp-p, @ericsnowcurrently, @serhiy-storchaka, @dianaclarke, @AraHaan, @jarondl |
| PRs |
bpo-25711: zipimport rewritten in Python #4023bpo-25711: Rewrite zipimport in pure Python. #6809bpo-25711: Remove outdated zipimport tests. #9404bpo-25711: Move _ZipImportResourceReader from importlib to zipimport. #9406bpo-32075: Expose ZipImporter Type Object in the include header files. #4470 |
| Files |
zipimport.patchzipimport-2.patch |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-11-05.13:48:52.968>
created_at = <Date 2015-11-23.16:28:40.454>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Rewrite zipimport from scratch'
updated_at = <Date 2018-11-05.13:48:52.967>
user = 'https://github.com/brettcannon' bugs.python.org fields:
activity = <Date 2018-11-05.13:48:52.967>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-05.13:48:52.968>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-11-23.16:28:40.454>
creator = 'brett.cannon'
dependencies = []
files = ['45800', '45814']
hgrepos = []
issue_num = 25711
keywords = ['patch']
message_count = 33.0
messages = ['255183', '255198', '255202', '255224', '256161', '256171', '256207', '257360', '257609', '282731', '282776', '282777', '282871', '282876', '282938', '283130', '299705', '304484', '304531', '306556', '306610', '307524', '316536', '321267', '321289', '323750', '324445', '324453', '324456', '325672', '325706', '325771', '329294']
nosy_count = 13.0
nosy_names = ['twouters', 'barry', 'brett.cannon', 'gregory.p.smith', 'ncoghlan', 'pmpp', 'eric.snow', 'serhiy.storchaka', 'diana', 'superluser', 'byrnes', 'Decorater', 'jarondl']
pr_nums = ['4023', '6809', '9404', '9406', '4470']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25711'
versions = ['Python 3.8'] Activitybrettcannon commented on Nov 23, 2015 No one wants to work on zipimport, and yet it's full of bugs. It needs a rewrite so that it's more maintainable. An idea floated at PyCon 2015 was to writing the zip-reading code in C and to keep it as simple as possible -- e.g., don't worry about supporting comments, etc. -- and then write the rest of the code in importlib, making maintenance much easier.
All of the various zipimport bugs should be made dependent on this issue as unless they are critical flaws I doubt they will get fixed without the rewrite. serhiy-storchaka commented on Nov 23, 2015 FYI I'm at the early stage of rewriting zipimport in Python. brettcannon commented on Nov 23, 2015 Are you writing it in such a way that it can be bootstspped in with importlib so the stslib can be loaded from it? serhiy-storchaka commented on Nov 23, 2015 superluser commented on Dec 9, 2015 Serhiy, how far along are you on this? I have a wip from this summer that I could finish over the holidays. serhiy-storchaka commented on Dec 10, 2015 I were on very early stage, and stopped this work few weeks ago in favor of other issues. I would be glad to make a review of your work when you have finished it Rose. vstinner commented on Dec 11, 2015 Can you both publish your WIP work? brettcannon commented on Jan 2, 2016 What are people's statuses on their various attempts? Since this is going to block my importlib.resources work I will do the work myself or work directly with someone in order to make sure this gets done. superluser commented on Jan 6, 2016 Sorry for the late response. I didn't have much time over the holidays. I think I better let someone else take this one. serhiy-storchaka commented on Dec 8, 2016 Here is preliminary translation of zipimport to Python. It is not frozen and imports other modules. I tried to keep the implementation close to C implementation. As a consequence, some raised exceptions look arbitrary. serhiy-storchaka commented on Dec 9, 2016 Got rid of dependencies from os, stat and encodings. vstinner commented on Dec 9, 2016
Here is preliminary translation of zipimport to Python. It is not frozen and imports other modules.
Technically, will it be possible to freeze it? It seems useful to keep the ability to put the whole stdlib into a single ZIP. Using a ZIP is sometimes suggested to avoid fstat() on disk for example, to speedup Python startup.
But I also understand that the C code is painful to maintain and update.
Anyway, nice job Serhiy :-) serhiy-storchaka commented on May 14, 2018 PR 6809 freezes zipimport.py, adds changes made in zipimport.c since writing the initial Python implementation, and fixes few bugs exposed in the frozen module. Seems Python now works with the zipped stdlib. serhiy-storchaka commented on Jul 8, 2018 Could anybody please make a review? This is just a first step, we need to do it before making other steps: implementing more modern import API, supporting large ZIP files and ZIP files with comments, implementing loading binary extensions from ZIP files, etc. brettcannon commented on Jul 8, 2018 I'm planning to review the PR at some point.
On Sun, Jul 8, 2018, 00:42 Serhiy Storchaka, <[email protected]> wrote:
Serhiy Storchaka <[email protected]> added the comment:
Could anybody please make a review? This is just a first step, we need to
do it before making other steps: implementing more modern import API,
supporting large ZIP files and ZIP files with comments, implementing
loading binary extensions from ZIP files, etc.
----------
Python tracker <[email protected]>
<https://bugs.python.org/issue25711\>
serhiy-storchaka commented on Aug 19, 2018 brettcannon commented on Aug 31, 2018 I think Serhiy's PR is basically done, so now the question is do we want to merge it in and drop the C code? ;)
I obviously say yes because this is I/O-bound code and so the switch shouldn't be enough of a performance hit to warrant blocking the gain in maintainability long-term (especially if we try to clean the module up slowly).
Barry, since I know you work with zip files a lot at work did you want to check to make sure perf won't be an issue for your use-case? warsaw commented on Sep 1, 2018 Not sure if I'll have time before the core sprints to check the implementation with shiv, but I can give it a try then. Do you mind waiting until then? brettcannon commented on Sep 1, 2018 I'm personally in no rush and I assume Serhiy isn't either with 3.8 cut-off quite a ways out, so I see no rush. serhiy-storchaka commented on Sep 18, 2018 serhiy-storchaka commented on Sep 19, 2018 serhiy-storchaka commented on Sep 19, 2018 ncoghlan commented on Nov 5, 2018 Noticed this was still open when reviewing Elvis's zipimport patch for bpo-34022.
Given the Python implementation has been merged, should we close this as resolved, and open new issues for any further changes (performance or otherwise)?
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|