GeezerVison

All commit authors signed the Contributor License Agreement.

I'm not quite sure if test_compression_supported_flags is needed here. The wrappers already import the extension modules at the top level, so this test can hardly fail. For example, the zlib subtest just compares the result of import zlib with itself.

Thanks for the review @aisk, both applied.

One open question: I left off @support.cpython_only. Nearly all existing ensure_lazy_imports call sites carry it (test_argparse.py is the exception), so happy to add it back if that's the preference :)

I guess it's the behavior of don't add a placeholder object to the module, when lazy importing, is the implement detail of CPython, so added the decorator to let other implementation to skip this.

In this PR, we don't use lazy import, so we should not add it.

Thanks @aisk leaving it as is then! Let me know if you have other comments

cc @emmatyping who may be interested in this.

Another idea, replace the three try/excepts with:

from importlib.util import find_spec

_BZ2_SUPPORTED = find_spec('_bz2') is not None
_LZMA_SUPPORTED = find_spec('_lzma') is not None
_ZSTD_SUPPORTED = find_spec('_zstd') is not None

Gives:

Before: 8 ms	PR: 6 ms	find_spec: 5 ms

Thanks @hugovk! I did try that first, and it's what pushed me to the import probe. As fas as I understandfind_spec answers a different question: it asks whether a module can be located, not whether it can be initialised. For an extension module those come apart, and when they do find_spec reports success where the import fails.

find_spec('_bz2') returns a spec as soon as ExtensionFileLoader finds the .so on the path. Everything that happens after that: dlopen, symbol resolution, PyInit__bz2, module exec, is where a broken extension actually fails, and find_spec never gets there.

Let's say the extension is compiled against a library that isn't present at runtime (ie: missing/mismatched libbz2). The .so is on disk, dlopen fails:

find_spec('_bz2')  -> True
import _bz2        -> ImportError: libbz2.so.1: cannot open shared object file

Let me know if I'm missing something, happy to use find_spec if you think that scenario can not happen but it might feels a bit safer to use import?

Ah yes, it is a change in behaviour. With main/this PR if the .so is corrupt, and we try and use it, we get:

ValueError: unknown archive format 'bztar'

With find_spec:

tarfile.CompressionError: bz2 module is not available

Which is maybe more informative? Anyway, it's probably not worth changing the behaviour for this small extra speedup, especially as shutil.get_archive_formats() then also claims "bztar"is available.

Thanks for confirming!

@hugo @aisk Let me know if you have any remaining feedback! I would love to see that landing in 3.15 if possible :) let me know!

I would love to see that landing in 3.15 if possible

The feature freeze has passed, so it will land only in 3.16.

ok! Thanks for confirming @StanFromIreland :)