GeezerVison is showing github.com — Detect QEMU linux-user emulation in `subprocess._use_posix_spawn()` · Issue #131398 · python/cpython
Detect QEMU linux-user emulation in subprocess._use_posix_spawn() #131398DescriptionThe qemu linux-user emulation does not support CLONE_VFORK and emulates it
with a true fork. That breaks posix_spawn because it cannot report errors
back to the parent process. The subprocess modules should detect that is
it being executed in qemu linux-user emulation and return False in
_use_posix_spawn. One way to do that is to look for "uarch *: qemu" in
/proc/cpuinfo. Activitychanged the title [-]Do not use posix_spawn in qemu linux-user emulation[/-] [+]Detect QEMU linux-user emulation in `subprocess._use_posix_spawn()`[/+] on Mar 18, 2025vstinner commented on Mar 20, 2025 How is it an issue to use vfork() if it's emulated with fork()? Is it a performance issue? Do you have examples of benchmark showing the performance problem? andreas-schwab commented on Mar 20, 2025 ContributorAuthor More actions vstinner commented on Mar 20, 2025 vstinner commented on Mar 20, 2025
Without a true vfork, the posix_spawn implementation in glibc cannot report a proper errno value for errors that happen in the child before execve.
Do you have an example to reproduce the issue? andreas-schwab commented on Mar 20, 2025 ContributorAuthor More actions It causes a failure while building the MozillaFirefox package:
4:55.15 Traceback (most recent call last):
4:55.16 File "/home/abuild/rpmbuild/BUILD/MozillaFirefox-136.0.1-build/firefox-136.0.1/security/nss/./coreconf/werror.py", line 80, in
4:55.16 main()
4:55.16 ~~~~^^
4:55.16 File "/home/abuild/rpmbuild/BUILD/MozillaFirefox-136.0.1-build/firefox-136.0.1/security/nss/./coreconf/werror.py", line 10, in main
4:55.16 cc_is_clang = 'clang' in subprocess.check_output(
4:55.16 ~~~~~~~~~~~~~~~~~~~~~~~^
4:55.16 [cc, '--version'], universal_newlines=True, stderr=sink)
4:55.16 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4:55.16 File "/usr/lib64/python3.13/subprocess.py", line 474, in check_output
4:55.16 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
4:55.16 ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4:55.17 **kwargs).stdout
4:55.17 ^^^^^^^^^
4:55.17 File "/usr/lib64/python3.13/subprocess.py", line 579, in run
4:55.17 raise CalledProcessError(retcode, process.args,
4:55.17 output=stdout, stderr=stderr)
4:55.17 subprocess.CalledProcessError: Command '['/usr/bin/ccache /usr/bin/gcc', '--version']' returned non-zero exit status 127.
The issue here is that posix_spawn should have set errno to ENOENT.
Also, the python testsuite tests this in test_subprocess.py. andreas-schwab commented on Mar 20, 2025 ContributorAuthor More actions For exmaple:
FAIL: test_exception_bad_args_0 (test.test_subprocess.POSIXProcessTestCase.test_exception_bad_args_0)
Test error in the child raised in the parent for a bad args[0].
Traceback (most recent call last):
File "/home/abuild/rpmbuild/BUILD/python313-3.13.2-build/Python-3.13.2/Lib/test/test_subprocess.py", line 1903, in test_exception_bad_args_0
self.fail("Expected OSError: %s" % desired_exception)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Expected OSError: [Errno 2] No such file or directory: '/_this/pa.th/does/not/exist' gpshead commented on Mar 20, 2025 FWIW I agreee with the approach of detecting this unusual/incomplete environment and having _use_posix_spawn() return False there. vfazio commented on Apr 7, 2025 Seems somewhat related: #129204. I ran into a similar scenario until i bumped QEMU to v7.2+ vfazio commented on Apr 7, 2025 Might also be worthwhile to not use posix_spawn if proc isn't mounted. As mentioned in the issue I linked, QEMU is sometimes run without proc or sys mounted due to limitations of the container environment so you may not even be able to query if you're running under emulation, in which case it may make sense to just assume we are.
For now, there's at least a knob (#132184) that can be exported into the environment to force the code to not use posix_spawn. gpshead commented on Apr 7, 2025 Saving some clicks, the undocumented knob added for CPython >=3.13.3 and >=3.14 is to set _PYTHON_SUBPROCESS_USE_POSIX_SPAWN=0 in your environment when you have an unusual environment that must not use it.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|