There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO python -m test should be enough to fit all use cases.
Maybe we need a --ci, --buildbot, or --strict option which would run tests is "strict" mode.
Portable way to run tests:
./python -m test
./python -m test.regrtest
./python -m test.libregrtest
./python -m test.autotest
./python Lib/test/regrtest.py
./python Lib/test/autotest.py
./python -c 'from test import autotest'
The main drawback is that running ./python doesn't work when Python is built with ./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ./python.exe must be run instead. On Windows, it's just python.
Unix, from Python source code:
make buildbottest TESTOPTS="..."
make test
make testall
make testuniversal
make hostrunnertest
./python Tools/scripts/run_tests.py
Windows, from Python source code:
PCbuild\rt.bat
Tools\buildbot\test.bat
-
Lib/test/autotest.py just runs test.libregrtest.main().
-
Lib/test/regrtest.py runs test.libregrtest.main() and remove Lib/test/ from sys.path and make __file__ an absolute path.
-
PCbuild\rt.bat:
- python: pass
-u -Wd -E -bb options
- regrtest: pass options passed to
rt.bat (except of options specific to rt.bat)
- most options are to get the path to the python.exe program in PCbuild/
-
Tools\buildbot\test.bat runs PCbuild\rt.bat:
rt.bat: pass -q -d options.- regrtest: pass
-j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
- ARM32 SSH pass
-unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.
-
make buildbottest runs Tools/scripts/run_tests.py:
- python: pass
$(TESTPYTHONOPTS) options.
- regrtest: pass
-j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.
-
Tools/scripts/run_tests.py:
- python: pass
-u -W default -bb -E options -- also cross-compilation options, not detailed here
- regrtest: pass
-r -w -j0 -u all,-largefile,-audio,-gui options, add -n on Windows.
GitHub Action workflow uses, .github/workflows/build.yml:
- Windows x86:
.\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
- Windows x64:
.\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
- macOS:
make buildbottest TESTOPTS="-j4 -uall,-cpu"
- Ubuntu:
xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
- Address Sanitizer:
xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
Buildbot use, master/custom/factories.py:
- UnixBuild and UnixCrossBuild:
make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
- UnixInstalledBuild:
/path/to/python -m test ... with code to select -m test options.
- BaseWindowsBuild:
Tools\buildbot\test.bat ... with code to select .... options
Linked PRs