Description

serhiy-storchaka

opened on Sep 25, 2023

Issue body actions

Feature or enhancement

Proposal:

libregrtest can run every test module in a separate process, and can even run several test modules in parallel. It has options to simply output all test modules or all test cases.

I propose to add an option to run every test case in a separate process. Test cases from the same module should still run sequentially, because they can use the same resource (for example create a server at fixed port).

This will slow down the tests several times, but it is not for regular running in the CI, it is for manual running once every months or for manual running selected test module while writing new tests or debugging test flakiness. It can help to solve the following issues:

1. Order dependency. Some tests are only passed because they run in specific order, and the order of tests in a module is the same, only the order of test modules is randomized. If run the test in a separate process, it is always the first, and cannot depend on others.
2. Environment changes and leaks. Some tests change the environment (e.g. leak temporary files or do not wait for terminating a thread or a subprocess), and it is not noticed because subsequent tests have cleaned after it. If run the test in a separate process, it is always the last, and nobody cleans after it.

cc @vstinner @gpshead

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-109817: Add --single-process-per-case option to libregrtest #151689
• gh-109817: Do not copy all test cases into the worker process #155713

Activity

1. serhiy-storchaka

   added

   type-featureA feature request or enhancementA feature request or enhancement

   testsTests in the Lib/test dirTests in the Lib/test dir

   on Sep 25, 2023
2. serhiy-storchaka commented on Sep 25, 2023

   

   serhiy-storchaka

   on Sep 25, 2023

   MemberAuthor

   More actions

   For now I can do this by using the following command:

   PYTHON=./python; $PYTHON -m test -uall --list-tests 2>/dev/null | while read -r test
   do
       $PYTHON -m test -uall --list-cases $test 2>/dev/null | xargs -n1 $PYTHON -m test -vuall $test -m
   done

   But it is slow (because it does not parallel different test modules), the output is difficult to analyze, and it cannot be run on Windows.
3. vstinner commented on Sep 25, 2023

   

   vstinner

   on Sep 25, 2023

   Member

   More actions

   See also closed PR gh-99637 by @gpshead.

   Reacted by Aniket
4. Aniketsy commented on Jun 5, 2026

   

   Aniketsy

   on Jun 5, 2026

   Contributor

   More actions

   hii @serhiy-storchaka i'm planning to go with this, please let me know if you have any pointers for me. currently going through the mentioned PR.
5. serhiy-storchaka commented on Jun 6, 2026

   

   serhiy-storchaka

   on Jun 6, 2026

   MemberAuthor

   More actions

   Look at the implementation of --list-tests and --list-cases options. What is common and what is different. Then look how running tests in multiple processes is implemented (option -j). Make it running tests one-case-per-process instead of one-module-per-process.

   This can be an easy task, but it can also be difficult. Only undertake this if you are confident in your abilities.

   Reacted by Aniket
6. serhiy-storchaka

   added a commit that references this issue on Aug 13, 2026

   gh-109817: Add --single-process-per-case option to libregrtest (GH-15…

   296f016
7. serhiy-storchaka

   closed this as completedin #151689on Aug 13, 2026
8. serhiy-storchaka

   reopened this on Aug 13, 2026
9. Aniketsy commented on Aug 13, 2026

   

   Aniketsy

   on Aug 13, 2026

   Contributor

   More actions

   @serhiy-storchaka Are we left with something in this or any improvement, please feel free to ping me . If this needs work
10. serhiy-storchaka

    added a commit that references this issue on Aug 13, 2026

    gh-109817: Do not copy all test cases into the worker process (GH-155713

    dd1194e
11. serhiy-storchaka commented on Aug 13, 2026

    

    serhiy-storchaka

    on Aug 13, 2026

    MemberAuthor

    More actions

    Running the whole test suite with --single-process-per-case found five pre-existing test bugs:

    • test_sysconfig leaves the sysconfig prefix cache modified #155726 -- test_sysconfig leaves the sysconfig prefix cache modified
    • test_import leaves files in the current directory #155731 -- test_import leaves files in the current directory
    • test_repr_lock leaves a dangling process #155732 -- test_repr_lock leaves a dangling process
    • test_attribute_completion fails because of a lazy internal import #155727 -- test_attribute_completion fails because of a lazy internal import
    • test_track_literals depends on a garbage collection #155728 -- test_track_literals depends on a garbage collection

    The first three modify the environment, but this is not detected when the environment is checked once per test file, because other tests in the same file clean up after them.

    It also made the lists of failed and skipped tests unreadable, because test case identifiers are longer than module names (GH-155714).

    Reacted by Aniket

    Reacted by Victor Stinner

    Reacted by Aniket

    Reacted by Victor Stinner
12. mbeijen

    added 2 commits that reference this issue on Aug 14, 2026

    pythongh-109817: Add --single-process-per-case option to libregrtest (p…

    7d9a9ef

    pythongh-109817: Do not copy all test cases into the worker process (p…

    8b1727c

Issue actions