GeezerVison

My hunch that nobody used to run a full suite with coverage seems to be corroborated by it not working, apparently. I was unable to get a full run of python -m test -T to complete on any version of Python between 3.8 and 3.13 on either Linux aarch64 or macOS M1.

So instead, I focused on bringing down the number of tests failing under coverage. With the latest change, tests already marked with @no_tracing are now temporarily disabling sys.monitoring events as well. This leaves 10 failing tests, all of which also failed with coverage before this PR. I will be fixing those in separate PRs because that's not related to the change in this PR.

10 tests failed:
    test_capi test_curses test_descr test_dis test_frame test_gc
    test_monitoring test_opcache test_tracemalloc test_types

In any case, unless anybody's got any strong opinions on this, this is ready to land.

I would prefer to also use test.cov when tests are run sequentially. Here is a patch to support it as well. It uses the new libregrtest code to "add options to Python":

diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py
index a5f02d6335..cf3943dd5d 100644
--- a/Lib/test/libregrtest/cmdline.py
+++ b/Lib/test/libregrtest/cmdline.py
@@ -426,8 +426,6 @@ def _parse_args(args, **kwargs):
             ns.rerun = True
         ns.print_slow = True
         ns.verbose3 = True
-    else:
-        ns._add_python_opts = False
 
     # When both --slow-ci and --fast-ci options are present,
     # --slow-ci has the priority
@@ -448,16 +446,8 @@ def _parse_args(args, **kwargs):
 
     if ns.single and ns.fromfile:
         parser.error("-s and -f don't go together!")
-    if ns.trace:
-        if ns.use_mp is not None:
-            if not Py_DEBUG:
-                parser.error("need --with-pydebug to use -T and -j together")
-        else:
-            print(
-                "Warning: collecting coverage without -j is imprecise. Configure"
-                " --with-pydebug and run -m test -T -j for best results.",
-                file=sys.stderr
-            )
+    if ns.trace and not Py_DEBUG:
+        parser.error("need --with-pydebug to use -T")
     if ns.python is not None:
         if ns.use_mp is None:
             parser.error("-p requires -j!")
diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index 296e1b4631..2a16969814 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -605,6 +605,8 @@ def _add_python_opts(self):
         environ, keep_environ = self._add_cross_compile_opts(regrtest_opts)
         if self.ci_mode:
             self._add_ci_python_opts(python_opts, keep_environ)
+        if self.coverage:
+            python_opts.extend(('-X', 'test.cov'))
 
         if (not python_opts) and (not regrtest_opts) and (environ is None):
             # Nothing changed: nothing to do

This leaves 10 failing tests, all of which also failed with coverage before this PR. I will be fixing those in separate PRs because that's not related to the change in this PR.

I suppose that a function can be added to test.support to check if code coverage is enabled, so we can skip lines of these tests which are impacted by coverage. Extract of test_descr:

            new_objects = len(gc.get_objects())
            self.assertEqual(orig_objects, new_objects)

Well, if a tracing tool runs in the background, such test can fail. So yeah, we should skip these lines, or just the whole test method.

Thanks for the review!

I'd prefer to change sequential -T runs to sys.monitoring in a future update because it's not as simple as your quick-and-dirty patch. By the time you add opts in _add_python_opts, libregrtest already imported a ton of libraries, the data is already ruined. Running -T sequentially with sys.monitoring will require -Xpresite= to be passed manually by the user, which I would prefer people don't have to do. So we would have to change sequential tests to execve or something with -Xpresite=.

Then there's the complication that -Xpresite is only available for Py_DEBUG builds because there were concerns this new hook could be misused by users. We can revisit adding it to production builds in the future but I don't want to block this PR on that discussion.

By the time you add opts in _add_python_opts, libregrtest already imported a ton of libraries, the data is already ruined.

This function replaces the current process with a new process which has the expected Python options: see _execute_python() method. On Unix, it uses the magic os.execv() function.

Then there's the complication that -Xpresite is only available for Py_DEBUG builds because there were concerns this new hook could be misused by users. We can revisit adding it to production builds in the future but I don't want to block this PR on that discussion.

It makes sense, ok. I don't want to block the PR, I just mean that "it would be nice" to use test.cov in all cases.

Thanks for reviews, Victor and Serhiy! ✨ 🍰 ✨