Description

serhiy-storchaka

opened on Sep 2, 2023

Issue body actions

Feature or enhancement

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Proposal:

I propose to add in libregrtest the support of selecting tests not only by names, but also by labels. Some labels are set when the test is decorated with corresponding decorator, others can be set by user manually.

For example, "requires_cpu" will mark tests decorated with @requires_resouce('cpu'), "impl_detail_cpython" will mark tests decorated with @cpython_only, "bigmemtest" will mark tests decorated with @bigmemtest(...). Tests related to pickling can be manually marked with label "pickletest".

Added two options --label=NAME for including only tests with the specified label, --no-label=NAME for excluding tests with the specified label.

You can request all test cases with the specified label using option --list-cases. You can run only tests with or without the specified label.

Added function mark() in the test.support module for manual marking tests.

@test.support.mark('pickletest')
def test_pickling(self):

Added also functions skipIf() and skipUnless() with additional argument specifying the label. It is for easier creation of custom decorators, e.g.

requires_foo = test.support.skipUnless(has_foo, 'requires Foo', label='requires_foo')

If you simply decorate the test method or class, you can just combine decorators:

@unittest.skipUnless(has_foo, 'requires Foo')
@test.support.mark('requires_foo')
def test_with_foo(self):

Adding the label to the test class is equivalent to adding the label to every method of that class.

Currently it only works with decorated test classes and test methods. If the whole module is skipped by using requires('gui'), classes and tests in that module cannot be found by the "requires_gui" label. If the test requires, for example, the "network" resource, but calls @requires('network') instead of been decorated with @requires_resource('network'), it cannot be found by the "requires_network" label.

It is an experimental feature, and can be changed in future (for example support labels with values, support glob patterns for labels). I think that in future some version of this feature will be added in unittest.

Linked PRs

• gh-108828: Support selecting tests by labels #108829

Reacted by Gregory P. Smith

Activity

1. serhiy-storchaka

   added

   type-featureA feature request or enhancementA feature request or enhancement

   testsTests in the Lib/test dirTests in the Lib/test dir

   on Sep 2, 2023
2. serhiy-storchaka

   added a commit that references this issue on Sep 2, 2023

   pythongh-108828: Support selecting tests by labels

   9aad8b0
3. serhiy-storchaka commented on Sep 3, 2023

   

   serhiy-storchaka

   on Sep 3, 2023

   MemberAuthor

   More actions

   Added support of labels of modules. Now --label requires_gui finds a lot of Tkinter and IDLE tests.
4. vstinner commented on Sep 4, 2023

   

   vstinner

   on Sep 4, 2023

   Member

   More actions

   I don't understand the use cases of this feature. Can you please give some concrete examples?

   I'm not sure that I would use such feature. Is it to count tests of a specific label to produce statistics? Or to only run tests of a label?

   For requires_cpu tests, we have buildbots running them. If we have no automated infra running a test frequently, maybe it's better to give up and even remove the test.

   At least, I managed to use it :-)

   $ ./python -m test -v test_tarfile --list-cases|wc -l
   588
   $ ./python -m test -v test_tarfile --label requires_gzip --list-cases|wc -l
   98
   

   So I see that the majority of test_tarfile tests don't use gzip, and 98 of them need gzip.
5. serhiy-storchaka commented on Sep 5, 2023

   

   serhiy-storchaka

   on Sep 5, 2023

   MemberAuthor

   More actions

   It is both for counting and for running.

   There was a complain that we may skip too many tests without knowing how much of tests were skipped and for what reasons. Your counting feature answered the part of this question. This issue is another answer, from other angle. I myself wanted to see how many tests will be marked with requires_resource('walltime') and run them sequentially before merging #108480. If in future we optimize tests we can found that some of them are fast enough and no need such decoration.

   Other example -- bigmem tests. They are usually skipped on buildbot, but it is useful to run them manually from time to time. They cannot be run in parallel, you need to run them sequentially, so it takes a long time to run them together with other tests, and their skip messages are lost in reports about successful pass of other tests. This feature allows: 1) to get a list of all bigmem tests which are scattered across many files to run them separately, 2) to run only bigmem tests directly (unfortunately there is still many noise and overhead added).

   Other example -- when working on some pickle changes I want to run pickling related tests which are scattered across many files multiple times. Running all tests takes too many time (well, it is now less, after excluding the slowest tests, but still too much to simply stare at screen while they run).

   Other example -- when working on some general C API changes I wanted to run all C API tests, which also were scattered across many files.

   Negative label matching can be used to search tests for manual marking. For example, --match '*ickle*' --no-label pickletest will give a list of candidates for @mark('pickletest') which still do not have this decorator.
6. serhiy-storchaka

   added a commit that references this issue on Sep 11, 2023

   pythongh-108828: Support selecting tests by labels

   6ea2f47
7. vstinner commented on Feb 27, 2025

   

   vstinner

   on Feb 27, 2025

   Member

   More actions

   What's the status of this issue?

Issue actions