GeezerVison is showing github.com — test_interpreters: test_create_many_threaded() failed on FreeBSD: log: RuntimeError: interpreter creation failed · Issue #109700 · python/cpython
test_interpreters: test_create_many_threaded() failed on FreeBSD: log: RuntimeError: interpreter creation failed #109700Descriptiontest_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ...
Warning -- Uncaught thread exception: RuntimeError
Exception in thread Thread-83 (task):
RuntimeError: error evaluating path
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 1059, in _bootstrap_inner
self.run()
File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/threading.py", line 996, in run
self._target(*self._args, **self._kwargs)
File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/test_interpreters.py", line 483, in task
interp = interpreters.create()
^^^^^^^^^^^^^^^^^^^^^
File "/buildbot/buildarea/3.x.ware-freebsd/build/Lib/test/support/interpreters.py", line 25, in create
id = _interpreters.create(isolated=isolated)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: interpreter creation failed
ok
build: https://buildbot.python.org/all/#/builders/1223/builds/187
Linked PRs
Activityvstinner commented on Sep 21, 2023 serhiy-storchaka commented on Sep 26, 2023 Important part in output:
0:05:34 load avg: 1.66 [424/463/1] test_interpreters failed (env changed) -- running (1): test_threading (2 min 42 sec)
Exception ignored error evaluating path:
Traceback (most recent call last):
File "<frozen getpath>", line 356, in <module>
ValueError: embedded null byte
vstinner commented on Sep 26, 2023
ValueError: embedded null byte
It comes from the FreeBSD build.
The aarch64 RHEL8 LTO + PGO 3.x build has a different error: TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object.
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
isolated = 0
environment = 0
user site = 1
safe_path = 0
import site = 1
is in build tree = 1
stdlib dir = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/Lib'
sys._base_executable = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
sys.base_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
sys.base_exec_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
sys.platlibdir = 'lib'
sys.executable = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/python'
sys.prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
sys.exec_prefix = '/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target'
sys.path = [
'/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/target/lib/python313.zip',
'/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/Lib',
'/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/lib.linux-aarch64-3.13',
]
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
File "<frozen importlib._bootstrap_external>", line 1100, in get_code
File "<frozen importlib._bootstrap_external>", line 1199, in get_data
TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object
vstinner commented on Sep 26, 2023 I failed to reproduce the issue on Linux just with this command:
./python -m test test_interpreters -v --forever -j25 --fail-env-changed
serhiy-storchaka commented on Sep 27, 2023
'/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/lib.linux-aarch64-3.13',
This looks very suspicious. Why two /build/s in a row? vstinner commented on Sep 27, 2023
This looks very suspicious. Why two /build/s in a row?
That's how aarch64 RHEL8 LTO + PGO 3.x is configured.
- Python source code:
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build
- Main test process run in:
/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto-pgo/build/build/test_python_worker_2806656æ
Yes, there are two build/build/ sub-directories, but I just think that it's a choice of the buildbot configuration, not a Python bug. serhiy-storchaka commented on Sep 27, 2023 It may be related to #109615. What if run tests locally in such configuration? serhiy-storchaka commented on Sep 27, 2023 This test consumes a lot of memory. 200 threads need more than 6 GB of memory if run them simultaneous (and if not, then what is the point of using so many threads?). Perhaps there is a leak, because if run tests repeatedly with limited memory, they finally crash.
$ (ulimit -v 7000000; ./python -m test -vuall test_interpreters -m test_create_many_threaded --forever)
...
0:00:11 load avg: 37.22 [ 5] test_interpreters
test_create_many_threaded (test.test_interpreters.StressTests.test_create_many_threaded) ... MemoryErrorMemoryErrorMemoryErrorFatal Python error: Segmentation fault
Current thread 0x00007fc450032640 (most recent call first):
<no Python frame>
200 live interpreters also consume sufficient amount of memory.
All normal tests only need 600-700 MB of memory. Tests which need more are marked with @bigmemtest decorator and perform a dry run or skipped by default. It seems that not all buildbots have so much memory.
@ericsnowcurrently What is the purpose of this test? Can it use less threads, for example 10 threads, each sequentially creating 20 interpreters? Can 5 threads be enough? serhiy-storchaka commented on Sep 27, 2023 test_create_many_sequential also crashes due to leaks.
$ (ulimit -v 700000; ./python -m test -vuall test_interpreters -m test_create_many_sequential --forever --fail-env-changed)
...
0:00:02 load avg: 0.62 [ 2] test_interpreters
test_create_many_sequential (test.test_interpreters.StressTests.test_create_many_sequential) ... Traceback (most recent call last):
File "/home/serhiy/py/cpython/Lib/site.py", line 73, in <module>
File "/home/serhiy/py/cpython/Lib/os.py", line 29, in <module>
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
File "<frozen importlib._bootstrap_external>", line 1137, in get_code
File "<frozen importlib._bootstrap_external>", line 766, in _compile_bytecode
Fatal Python error: Segmentation fault
Current thread 0x00007f60db746740 (most recent call first):
<no Python frame>
If you double the limit, it will crash on the 7th iteration instead of the 2nd. If triple -- on 11th. So it leaks 1.5-2 MB per interpreter. These two tests ran sequentially can leak up to 6 GB of memory. vstinner commented on Sep 27, 2023 Wow, so now it's possible to leak a whole interpreter?
Maybe we need something like threading_setup() / threading_cleanup() which uses _thread._count(), to count how many interpreters we have before/after running tests? serhiy-storchaka commented on Sep 28, 2023
These two tests ran sequentially can leak up to 6 GB of memory.
Actually, only up to 600 MB, but it is much anyway. I do not know how it is now, but several years ago some of buildbots had only few hundreds of MBs of physical memory, often failed after hours of swapping due to timeout. I suppose buildbots on which test_create_many_threaded fails also have very limited RAM. serhiy-storchaka commented on Sep 28, 2023 More precisely, 1.4 MB are leaked in every subinterpreter. vstinner commented on Sep 28, 2023
More precisely, 1.4 MB are leaked in every subinterpreter.
Almost a floppy disk (1.44 MB)! Reacted by Serhiy Storchaka, Alex Waygood and sunmy2019 kumaraditya303 commented on Oct 18, 2025 The issue should be fixed now with #140067 serhiy-storchaka commented on Oct 18, 2025
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|