GeezerVison is showing github.com — importlib.resources.files() doesn't work correctly when importlib library is compiled · Issue #123085 · python/cpython
importlib.resources.files() doesn't work correctly when importlib library is compiled #123085DescriptionBug report
Bug description:
If importlib library is compiled (in particular _common.py), then any package, that uses bare files() to get its resources, breaks, because _infer_caller returns wrong frame (as mentioned in #123037 (comment), __file__ in
|
return frame_info.filename == __file__ |
is not valid)
Reproducer:
import importlib
import os
import pathlib
import py_compile
import shutil
import sys
import tempfile
import textwrap
def compile(tempdir):
target_dir = pathlib.Path(tempdir) / 'cimportlib'
souce_dir = pathlib.Path(importlib.__file__).parent
shutil.copytree(souce_dir, target_dir, ignore=lambda *_: ['__pycache__'])
for dirpath, _, filenames in os.walk(target_dir):
for filename in filenames:
if filename != "_common.py":
continue
source_path = pathlib.Path(dirpath) / filename
cfile = source_path.with_suffix('.pyc')
py_compile.compile(source_path, cfile)
pathlib.Path.unlink(source_path)
def create_package(tempdir):
package_dir = pathlib.Path(tempdir) / 'somepkg'
package_dir.mkdir()
contents = {
"__init__.py": textwrap.dedent(
"""
import cimportlib.resources as res
val = res.files().joinpath('resource.txt').read_text(encoding='utf-8')
"""
),
"resource.txt": "data",
}
for file, content in contents.items():
path = pathlib.Path(package_dir) / file
path.write_text(content)
def main():
with tempfile.TemporaryDirectory() as tempdir:
compile(tempdir)
create_package(tempdir)
sys.path.insert(0, str(tempdir))
print(importlib.import_module('somepkg').val)
if __name__ == "__main__":
raise SystemExit(main())Expectation:
Actual outcome:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpngnuw441/cimportlib/resources/resource.txt' It may be important for frozen python applications, as they don't include source code
CPython versions tested on:
3.12, CPython main branch
Operating systems tested on:
Linux, Windows
Linked PRs
Activitychanged the title [-]importlib.files() doesn't work correctly when importlib library is compiled[/-] [+]importlib.resources.files() doesn't work correctly when importlib library is compiled[/+] on Aug 17, 2024jaraco commented on Aug 17, 2024
It may be important for frozen python applications
Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code? Gatsik commented on Aug 17, 2024
Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?
I encountered it in practice encukou commented on Sep 13, 2024 Some buildbots started failing when #123102 was merged, see:
I'll probably not have to investigate this week, but, my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”. jaraco commented on Sep 13, 2024 In the first log, I see:
0:01:29 load avg: 11.57 [1/1/1] test_importlib failed (2 errors)
Re-running test_importlib in verbose mode (matching: test_implicit_files_with_compiled_importlib, test_implicit_files_with_compiled_importlib)
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
self._compile_importlib()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
py_compile.compile(source_path, cfile)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmpw7y1yd3q/c_resources/abc.pyc.281473105247840'
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
self._compile_importlib()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
py_compile.compile(source_path, cfile)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmp52oghnez/c_resources/abc.pyc.281473106346832'
----------------------------------------------------------------------
Ran 2 tests in 0.020s
The error is the same on the other buildbot. Both are Fedora, so perhaps that's a factor.
It's failing on this open call:
|
fd = _os.open(path_tmp, |
|
_os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, mode & 0o666) |
It's conceivable the copy,
|
shutil.copytree(sources, c_resources, ignore=lambda *_: ['__pycache__']) |
while it excludes __pycache__ could still be copying extant .pyc files and those are failing to write atomically. Although, that's probably not it, because the failure is happening when writing out the temp file.
my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”.
I'm not familiar with any OS that has such behaviors. I'd not have considered it. I did read up a bit on the noexec mount option. At first blush, I wouldn't expect such an option to conflict with _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY.
I don't think I'll make much more progress on this without a local reproducer. Do you think this error can be reproduced in a Docker container? In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files? encukou commented on Sep 17, 2024
Do you think this error can be reproduced in a Docker container?
I couldn't reproduce it locally. It might need some specific VM settings :(
In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?
First let me try an implementation that avoids the initial copy, which should be a bit faster too: #124131 added a commit that references this issue on Sep 18, 2024gh-123085: _compile_importlib: Avoid copying sources before compilati… godlygeek commented on Aug 14, 2025 I've also encountered this issue in practice. It causes a failure when the stdlib is zipped, and
(importlib.resources.files() / "datafile.txt").read_text() fails with
FileNotFoundError: /opt/python/lib/python313.zip/importlib/resources/datafile.txt
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|