Running¶

The shortest, simplest way of running the test suite is the following command, which runs tests in parallel, from the root directory of your checkout (after you have built Python):

./python -m test -j0

./python.exe -m test -j0

.\python.bat -m test -j0

This will run the majority of tests, but exclude a small portion of them; these excluded tests use special kinds of resources: for example, accessing the Internet, or trying to play a sound or to display a graphical interface on your desktop. They are disabled by default so that running the test suite is not too intrusive. To enable some of these additional tests (and for other flags which can help debug various issues such as reference leaks), read the help text:

./python -m test -h

./python.exe -m test -h

.\python.bat -m test -h

If you want to run a single test file, simply specify the test file name (without the extension) as an argument. You also probably want to enable verbose mode (using -v), so that individual failures are detailed:

./python -m test -v test_abc

./python.exe -m test -v test_abc

.\python.bat -m test -v test_abc

To run a single test case, use the unittest module, providing the import path to the test case:

./python -m unittest -v test.test_abc.TestABC_Py

./python.exe -m unittest -v test.test_abc.TestABC_Py

.\python.bat -m unittest -v test.test_abc.TestABC_Py

Some test modules also support direct invocation, which might be useful for IDEs and local debugging:

./python Lib/test/test_typing.py

./python.exe Lib/test/test_typing.py

.\python.bat Lib/test/test_typing.py

But, there are several important notes:

1. This way of running tests exists only for local developer needs and is discouraged for anything else

2. Some modules do not support it at all. One example is test_importlib. In other words: if some module does not have unittest.main(), then most likely it does not support direct invocation.

By default, -j0 runs tests in parallel using a number of worker processes based on the CPU count, which is much faster but can interleave output from different tests. To run tests sequentially, for example when debugging a failure, drop the -j flag:

./python -m test

./python.exe -m test

.\python.bat -m test

To limit how many tests run at once, pass an explicit number of worker processes instead, such as -j2.

Finally, if you want to run tests under a more strenuous set of settings, you can run test as:

./python -bb -E -Wd -m test -r -w -uall

./python.exe -bb -E -Wd -m test -r -w -uall

.\python.bat -bb -E -Wd -m test -r -w -uall

The various extra flags passed to Python cause it to be much stricter about various things (the -Wd flag should be -W error at some point, but the test suite has not reached a point where all warnings have been dealt with and so we cannot guarantee that a bug-free Python will properly complete a test run with -W error). The -r flag to the test runner causes it to run tests in a more random order which helps to check that the various tests do not interfere with each other. The -w flag causes failing tests to be run again to see if the failures are transient or consistent. The -uall flag allows the use of all available resources so as to not skip tests requiring, for example, Internet access.

To check for reference leaks (only needed if you modified C code), you can enable reference leak checking during testing using the -R flag. For example, using the command:

python -m test <test_name> -R :

This default setting performs a few initial warm-up runs to stabilize the reference count, followed by additional runs to check for leaks.

If you want more control over the number of runs, you can specify warmups and repeats explicitly:

python -m test <test_name> -R <warmups>:<repeats>

For instance, -R 3:2 will first run the test 3 times to settle down the reference count, and then run it 2 more times to check for leaks.

You can also execute the Tools/scripts/run_tests.py script as found in a CPython checkout. The script tries to balance speed with thoroughness. But if you want the most thorough tests you should use the strenuous approach shown above.

Writing¶

Writing tests for Python is much like writing tests for your own code. Tests need to be thorough, fast, isolated, consistently repeatable, and as simple as possible. We try to have tests both for normal behaviour and for error conditions. Tests live in the Lib/test directory, where every file that includes tests has a test_ prefix.

One difference with ordinary testing is that you are encouraged to rely on the test.support module. It contains various helpers that are tailored to Python’s test suite and help smooth out common problems such as platform differences, resource consumption and cleanup, or warnings management. That module is not suitable for use outside of the standard library.

When you are adding tests to an existing test file, it is also recommended that you study the other tests in that file; it will teach you which precautions you have to take to make your tests robust and portable.

For tests of the C API, see Tests sections in Changing Python’s C API.

Benchmarks¶

Benchmarking is useful to test that a change does not degrade performance.

The Python Benchmark Suite has a collection of benchmarks for all Python implementations. Documentation about running the benchmarks is in the README.txt of the repo.