Description

encukou

opened on Oct 31, 2024

Issue body actions

Running the tests with --junit-xml, the result does not include subTest results, as RegressionTestResult in libregrtest.testresult does not implement addSubTest.

There's no standard XML schema for the test results (see testmoapp/junitxml), but I think multiple <error> elements in a <testcase> make sense

Reacted by Donghee Na

Activity

1. encukou

   added

   testsTests in the Lib/test dirTests in the Lib/test dir

   on Oct 31, 2024
2. sobolevn commented on Nov 1, 2024

   

   sobolevn

   on Nov 1, 2024

   Member

   More actions

   @encukou do you already work on this? Or can I take this over, please? :)
3. encukou commented on Nov 1, 2024

   

   encukou

   on Nov 1, 2024

   MemberAuthor

   More actions

   I just moved it out of my TODO list, thinking it might be fun for someone else :)
   If you want, please do take over!

   Reacted by sobolevn
4. sobolevn

   self-assigned this

   on Nov 1, 2024
5. sobolevn commented on Nov 3, 2024

   

   sobolevn

   on Nov 3, 2024

   Member

   More actions

   What happens now for a test like this:

       def test_broken_subtest(self):
           for x in [1, 2, 3]:
               with self.subTest(x=x):
                   self.assertEqual(x, 0)

   produces:

   <testsuites tests="1" errors="0" failures="3"><testsuite start="2024-11-03 09:35:14.984110" tests="1" errors="0" failures="3"><testcase /></testsuite></testsuites>

   If we comment # with self.subTest(x=x):, the output changes to:

   <testsuites tests="1" errors="0" failures="1"><testsuite start="2024-11-03 09:34:09.426455" tests="1" errors="0" failures="1"><testcase name="test.test_typing.TypeVarTests.test_broken_subtest" status="run" result="completed" time="0.002489"><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 1 != 0&#10;">Traceback (most recent call last):
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 660, in run
       self._callTestMethod(testMethod)
       ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 606, in _callTestMethod
       result = method()
     File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 569, in test_broken_subtest
       self.assertEqual(x, 0)
       ~~~~~~~~~~~~~~~~^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
       assertion_func(first, second, msg=msg)
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
       raise self.failureException(msg)
   AssertionError: 1 != 0
   </failure></testcase></testsuite></testsuites>

   Basically, we need to have multiple <failure> nodes inside <testcase> with .subTest calls.
6. sobolevn commented on Nov 3, 2024

   

   sobolevn

   on Nov 3, 2024

   Member

   More actions

   I don't quite like that there are multiple <failure> and multiple <system-out> and <system-err> tags:

   <testsuites tests="1" errors="0" failures="3"><testsuite start="2024-11-03 14:20:57.276683" tests="1" errors="0" failures="3"><testcase name="test.test_typing.TypeVarTests.test_broken_subtest" status="run" result="completed" time="0.004247"><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 1 != 0&#10;">Traceback (most recent call last):
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
       self.assertEqual(x, 0)
       ~~~~~~~~~~~~~~~~^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
       assertion_func(first, second, msg=msg)
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
       raise self.failureException(msg)
   AssertionError: 1 != 0
   </failure><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 2 != 0&#10;">Traceback (most recent call last):
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
       self.assertEqual(x, 0)
       ~~~~~~~~~~~~~~~~^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
       assertion_func(first, second, msg=msg)
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
       raise self.failureException(msg)
   AssertionError: 2 != 0
   </failure><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 3 != 0&#10;">Traceback (most recent call last):
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
       yield
     File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
       self.assertEqual(x, 0)
       ~~~~~~~~~~~~~~~~^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
       assertion_func(first, second, msg=msg)
       ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
     File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
       raise self.failureException(msg)
   AssertionError: 3 != 0
   </failure></testcase></testsuite></testsuites>

   I think it would be hard to parse. Not sure that it is even supported.
   See https://github.com/testmoapp/junitxml/blob/1bcb6fb8b620906ec001a2c8e7a7c8e7c563a64b/examples/junit-complete.xml#L99

Issue actions