GeezerVison is showing github.com — Can't open orphan path on bare importlib.resources.files() under zipapp · Issue #121735 · python/cpython
Can't open orphan path on bare importlib.resources.files() under zipapp #121735DescriptionBug report
Bug description:
Context: In Python 3.12 the function importlib.resources.files() was updated: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files
Changed in version 3.12: package parameter was renamed to anchor. anchor can now be a non-package module and if omitted will default to the caller’s module. package is still accepted for compatibility but will raise a DeprecationWarning. Consider passing the anchor positionally or using importlib_resources >= 5.10 for a compatible interface on older Pythons.
The Issue: When using importlib.resources.files(), I am able to open and read a text file included as package data in my package. However, if I zip up my package with zipapp and run the .pyz file, it is unable to read the text file.
If I use importlib.resources.files("mypkg") it will work in both cases.
Minimal example:
Create a project directory named mypkg with the src layout.
Create the following files.
mypkg/src/mypkg/a.py
import importlib.resources
def main():
x = importlib.resources.files() / "data.txt"
print(x.read_text())mypkg/src/mypkg/data.txt
In pyproject.toml make sure to set up the entry script and include the data file.
mypkg/pyproject.toml
[project.scripts]
myscript = "mypkg.a:main"
[tool.setuptools.package-data]
mypkg = ["*.txt"]
Install the package into a fresh Python (virtual) environment with pip.
Run myscript and verify it prints datadata.
Now back in the mypkg project root folder, we will generate a .pyz file with zipapp and run the program from the .pyz file.
Run pip install . --target buildtemp --upgrade && python3 -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env python3' -o mypkg.pyz in the terminal.
Run ./mypkg.pyz. This results in an error:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/__main__.py", line 3, in <module>
File "/home/win/code/sandbox/python/mypkg/./mypkg.pyz/mypkg/a.py", line 6, in main
File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/abc.py", line 89, in read_text
with self.open(encoding=encoding) as strm:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/win/.local/share/mise/installs/python/3.12.4/lib/python3.12/importlib/resources/_adapters.py", line 139, in open
raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path
Note that if you change x = importlib.resources.files() / "data.txt" to x = importlib.resources.files("mypkg") / "data.txt", it will work when running the .pyz file.
CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
Activitywimglenn commented on Aug 14, 2024 ping @jaraco in case you missed this one Reacted by Jason R. Coombs jaraco commented on Aug 14, 2024 The code that handles the module inference is here:
|
@functools.singledispatch |
|
def resolve(cand: Optional[Anchor]) -> types.ModuleType: |
|
return cast(types.ModuleType, cand) |
|
|
|
|
|
@resolve.register |
|
def _(cand: str) -> types.ModuleType: |
|
return importlib.import_module(cand) |
|
|
|
|
|
@resolve.register |
|
def _(cand: None) -> types.ModuleType: |
|
return resolve(_infer_caller().f_globals['__name__']) |
|
|
|
|
|
def _infer_caller(): |
|
""" |
|
Walk the stack and find the frame of the first caller not in this module. |
|
""" |
|
|
|
def is_this_file(frame_info): |
|
return frame_info.filename == __file__ |
|
|
|
def is_wrapper(frame_info): |
|
return frame_info.function == 'wrapper' |
|
|
|
not_this_file = itertools.filterfalse(is_this_file, inspect.stack()) |
|
# also exclude 'wrapper' due to singledispatch in the call stack |
|
callers = itertools.filterfalse(is_wrapper, not_this_file) |
|
return next(callers).frame |
Next step is to determine what expectation is missed when running under zipapp. changed the title [-]Cannot read package data file after opening with importlib.resources.files(anchor) and running with zipapp[/-] [+]Can't open orphan path on bare importlib.resources.files() under zipapp[/+] on Aug 14, 2024jaraco commented on Aug 14, 2024 I was able to replicate the issue without going through the wheel build process:
draft 🐚 mkdir -p buildtemp/mypkg
draft 🐚 cat > buildtemp/mypkg/a.py
import importlib.resources
def main():
x = importlib.resources.files() / "data.txt"
print(x.read_text())
draft 🐚 echo datadata > buildtemp/mypkg/data.txt
draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
mypkg.a.main()
~~~~~~~~~~~~^^
File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
print(x.read_text())
~~~~~~~~~~~^^
File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
with self.open(encoding=encoding) as strm:
~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan pathjaraco commented on Aug 14, 2024 The issue is that CompatibilityFiles is being activated for loading resources.
draft 🐚 py -m pdb mypkg.pyz
> /Users/jaraco/draft/mypkg.pyz/__main__.py(2)<module>()
-> import mypkg.a
(Pdb) c
Traceback (most recent call last):
File "/opt/python/lib/python3.13/pdb.py", line 2435, in main
pdb._run(target)
~~~~~~~~^^^^^^^^
File "/opt/python/lib/python3.13/pdb.py", line 2192, in _run
self.run(target.code)
~~~~~~~~^^^^^^^^^^^^^
File "/opt/python/lib/python3.13/bdb.py", line 651, in run
exec(cmd, globals, locals)
~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jaraco/draft/mypkg.pyz/__main__.py", line 3, in <module>
mypkg.a.main()
~~~~~~~~~~~~^^
File "/Users/jaraco/draft/mypkg.pyz/mypkg/a.py", line 6, in main
print(x.read_text())
~~~~~~~~~~~^^
File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text
with self.open(encoding=encoding) as strm:
~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open
raise FileNotFoundError("Can't open orphan path")
FileNotFoundError: Can't open orphan path
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /opt/python/lib/python3.13/importlib/resources/_adapters.py(139)open()
-> raise FileNotFoundError("Can't open orphan path")
(Pdb) u
> /opt/python/lib/python3.13/importlib/resources/abc.py(89)read_text()
-> with self.open(encoding=encoding) as strm:
(Pdb) u
> /Users/jaraco/draft/mypkg.pyz/mypkg/a.py(6)main()
-> print(x.read_text())
(Pdb) x
<importlib.resources._adapters.CompatibilityFiles.OrphanPath object at 0x41f4c584f90>
CompatibilityFiles gets employed when there's no proper resource provider found. I'll figure out why.
jaraco commented on Aug 14, 2024 It's clear that the bare call to files() is getting CompatibiltyFiles when the call with an anchor returns a proper MultiplexedPath:
(Pdb) importlib.resources.files()
<importlib.resources._adapters.CompatibilityFiles.SpecPath object at 0x41f4c595a10>
(Pdb) importlib.resources.files('mypkg')
MultiplexedPath('/Users/jaraco/draft/mypkg.pyz/mypkg/')
By changing the main() function to emit the inferred caller's name,
def main():
print(importlib.resources._common._infer_caller().f_globals['__name__'])I can confirm that the inferred name is correct when it emits mypkg.a.
Interestingly, if I then replace the main function with a static 'mypkg.a', the reported error occurs.
def main():
x = importlib.resources.files('mypkg.a') / "data.txt"
print(x.read_text())So the issue seems to be that files('mypkg') works as expected but files('mypkg.a') does not, and it's only coincidental that files() calls files('mypkg.a').
importlib.resources has a behavior that when given a module will resolve resources in the package of that module.
It seems that the zipimporter as attached to the module is not resolving a resource reader:
(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg.a')
(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg')
(Pdb) importlib.import_module('mypkg').__spec__.loader.get_resource_reader('mypkg')
<importlib.resources.readers.NamespaceReader object at 0x3df2e585410>I'm slightly surprised to see NamespaceReader and not a ZipReader. jaraco commented on Aug 14, 2024 It seems the issue traces to the logic in get_resource_reader:
|
def get_resource_reader(self, fullname): |
|
"""Return the ResourceReader for a package in a zip file. |
|
|
|
If 'fullname' is a package within the zip file, return the |
|
'ResourceReader' object for the package. Otherwise return None. |
|
""" |
|
try: |
|
if not self.is_package(fullname): |
|
return None |
|
except ZipImportError: |
|
return None |
|
from importlib.readers import ZipReader |
|
return ZipReader(self, fullname) |
When checking loader.is_package for mypkg.a, it returns False:
(Pdb) loader = importlib.import_module('mypkg.a').__spec__.loader
(Pdb) loader.is_package('mypkg.a')
FalseIn the early days of importlib.resources, all resources needed to be relative to a package (and not a module). Later, that restriction was relaxed to allow resources to be loaded relative to a module.
But it seems that get_resource_reader didn't get the corresponding treatment.
I've confirmed the same issue occurs when using importlib_resources:
draft [1] 🐚 cat buildtemp/mypkg/a.py
import importlib_resources
def main():
x = importlib_resources.files('mypkg.a') / "data.txt"
print(x.read_text())
draft 🐚 pip install -t buildtemp importlib_resources
Collecting importlib_resources
Using cached importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB)
Using cached importlib_resources-6.4.0-py3-none-any.whl (38 kB)
Installing collected packages: importlib_resources
Successfully installed importlib_resources-6.4.0
draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz
draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
...
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt
I suspect the resolution is going to be update get_resource_reader to accept a module.
But first things first, let's figure out why this use-case isn't covered by the tests. jaraco commented on Aug 14, 2024 As a check, if I remove the is_package check:
cpython main 🐚 git diff
diff --git a/Lib/zipimport.py b/Lib/zipimport.py
index f2724dd0268..c0a4abcd95d 100644
--- a/Lib/zipimport.py
+++ b/Lib/zipimport.py
@@ -256,16 +256,8 @@ def load_module(self, fullname):
def get_resource_reader(self, fullname):
- """Return the ResourceReader for a package in a zip file.
-
- If 'fullname' is a package within the zip file, return the
- 'ResourceReader' object for the package. Otherwise return None.
+ """Return the ResourceReader for a module in a zip file.
"""
- try:
- if not self.is_package(fullname):
- return None
- except ZipImportError:
- return None
from importlib.readers import ZipReader
return ZipReader(self, fullname)The routine then fails, but with a different error:
draft 🐚 ./mypkg.pyz
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module>
mypkg.a.main()
~~~~~~~~~~~~^^
File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main
print(x.read_text())
~~~~~~~~~~~^^
File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 426, in read_text
with self.open('r', encoding, *args, **kwargs) as strm:
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 391, in open
raise FileNotFoundError(self)
FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txtThat's encouraging, suggesting that the ZipReader is getting involved, but that it still doesn't honor resources adjacent to modules. jaraco commented on Aug 14, 2024 I started working on a test, seeking to extend these tests to zip files, but realized that the current test structure isn't easily amenable to extension to zip. In python/importlib_resources@3faf336, I've done some refactoring in order to leverage the zip fixtures for this purpose. jaraco commented on Dec 24, 2024
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|