GeezerVison is showing github.com — Python 3.13: test_xml_etree fails on expat 2.2.5 (RHEL8 / Rocky linux 8 / Ubuntu 22.04) · Issue #125067 · python/cpython
Python 3.13: test_xml_etree fails on expat 2.2.5 (RHEL8 / Rocky linux 8 / Ubuntu 22.04) #125067DescriptionBug report
Bug description:
I'm attempting to build Python 3.13.0 for Rocky Linux 8, and several tests are failing:
LD_LIBRARY_PATH=/usr/src/python ./python -m test --pgo --timeout=
Using random seed: 1341012311
0:00:00 load avg: 15.81 Run 44 tests sequentially in a single process
0:00:00 load avg: 15.81 [ 1/44] test_array
0:00:02 load avg: 14.70 [ 2/44] test_base64
0:00:02 load avg: 14.70 [ 3/44] test_binascii
0:00:03 load avg: 14.70 [ 4/44] test_binop
0:00:03 load avg: 14.70 [ 5/44] test_bisect
0:00:03 load avg: 14.70 [ 6/44] test_bytes
0:00:13 load avg: 12.83 [ 7/44] test_bz2
0:00:14 load avg: 12.83 [ 8/44] test_cmath
0:00:14 load avg: 12.83 [ 9/44] test_codecs
0:00:17 load avg: 12.04 [10/44] test_collections
0:00:21 load avg: 12.04 [11/44] test_complex
0:00:22 load avg: 11.32 [12/44] test_dataclasses
0:00:23 load avg: 11.32 [13/44] test_datetime
0:00:31 load avg: 10.57 [14/44] test_decimal
0:00:41 load avg: 9.25 [15/44] test_difflib
0:00:44 load avg: 8.67 [16/44] test_embed
0:01:01 load avg: 7.35 [17/44] test_float
0:01:01 load avg: 7.35 [18/44] test_fstring
0:01:06 load avg: 7.00 [19/44] test_functools
0:01:08 load avg: 6.68 [20/44] test_generators
test test_generators failed
0:01:08 load avg: 6.68 [21/44] test_hashlib -- test_generators failed (1 failure)
0:01:10 load avg: 6.68 [22/44] test_heapq
0:01:11 load avg: 6.68 [23/44] test_int
0:01:14 load avg: 6.47 [24/44] test_itertools
0:01:25 load avg: 6.16 [25/44] test_json
0:01:33 load avg: 5.52 [26/44] test_long
0:01:40 load avg: 5.31 [27/44] test_lzma
0:01:41 load avg: 5.31 [28/44] test_math
0:01:49 load avg: 5.04 [29/44] test_memoryview
0:01:49 load avg: 5.04 [30/44] test_operator
0:01:50 load avg: 5.04 [31/44] test_ordered_dict
0:01:52 load avg: 5.04 [32/44] test_patma
0:01:52 load avg: 4.79 [33/44] test_pickle
0:02:07 load avg: 4.10 [34/44] test_pprint
0:02:08 load avg: 4.10 [35/44] test_re
0:02:10 load avg: 4.10 [36/44] test_set
0:02:24 load avg: 4.68 [37/44] test_sqlite3
0:02:26 load avg: 4.68 [38/44] test_statistics
0:02:45 load avg: 6.26 [39/44] test_str
0:02:50 load avg: 6.32 [40/44] test_struct
0:02:52 load avg: 6.32 [41/44] test_tabnanny
0:02:53 load avg: 6.45 [42/44] test_time
0:02:55 load avg: 6.45 [43/44] test_xml_etree
test test_xml_etree failed
0:02:57 load avg: 6.45 [44/44] test_xml_etree_c -- test_xml_etree failed (3 failures)
test test_xml_etree_c failed
test_xml_etree_c failed (3 failures)
Total duration: 2 min 59 sec
Total tests: run=9,361 failures=7 skipped=204
Total test files: run=44/44 failed=3
These tests are required to succeed in order for --enable-optimizations to work as expected.
I am using the following build options:
./configure \
--enable-loadable-sqlite-extensions \
--enable-optimizations \
--enable-shared \
--with-lto \
--with-system-expat \
--with-ensurepip \
LDFLAGS=-Wl,-rpath=/usr/local/lib
3.9-3.12.6 do not exhibit this behavior when built with the same commands (aside from the Python version).
I'm curious whether there might be a regression for certain versions of expat, similar to #117187.
expat-devel version is expat-2.2.5-15.el8.
The specific test failures are:
======================================================================
FAIL: test_flush_reparse_deferral_disabled (test.test_xml_etree.XMLPullParserTest.test_flush_reparse_deferral_disabled)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1752, in test_flush_reparse_deferral_disabled
self.assert_event_tags(parser, [('start', 'doc')])
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
self.assertEqual([(action, elem.tag) for action, elem in events],
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
expected)
^^^^^^^^^
AssertionError: Lists differ: [] != [('start', 'doc')]
Second list contains 1 additional elements.
First extra element 0:
('start', 'doc')
- []
+ [('start', 'doc')]
======================================================================
FAIL: test_simple_xml_chunk_1 (test.test_xml_etree.XMLPullParserTest.test_simple_xml_chunk_1)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1508, in test_simple_xml_chunk_1
self.test_simple_xml(chunk_size=1, flush=True)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1496, in test_simple_xml
self.assert_event_tags(parser, [('end', 'element')])
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
self.assertEqual([(action, elem.tag) for action, elem in events],
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
expected)
^^^^^^^^^
AssertionError: Lists differ: [] != [('end', 'element')]
Second list contains 1 additional elements.
First extra element 0:
('end', 'element')
- []
+ [('end', 'element')]
======================================================================
FAIL: test_simple_xml_chunk_5 (test.test_xml_etree.XMLPullParserTest.test_simple_xml_chunk_5)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1511, in test_simple_xml_chunk_5
self.test_simple_xml(chunk_size=5, flush=True)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1496, in test_simple_xml
self.assert_event_tags(parser, [('end', 'element')])
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/python/Lib/test/test_xml_etree.py", line 1483, in assert_event_tags
self.assertEqual([(action, elem.tag) for action, elem in events],
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
expected)
^^^^^^^^^
AssertionError: Lists differ: [] != [('end', 'element')]
Second list contains 1 additional elements.
First extra element 0:
('end', 'element')
- []
+ [('end', 'element')]
----------------------------------------------------------------------
Ran 198 tests in 0.423s
FAILED (failures=3, skipped=6)
test test_xml_etree failed
test_xml_etree failed (3 failures)
== Tests result: FAILURE ==
1 test failed:
test_xml_etree
Total duration: 640 ms
Total tests: run=198 failures=3 skipped=6
Total test files: run=1/1 failed=1
Result: FAILURE
CPython versions tested on:
3.13
Operating systems tested on:
Linux Reacted by Ed Morley, Pablo Alexis Domínguez Grau and Clémence Lesné Activitychanged the title [-]Python 3.13[/-] [+]Python 3.13: tests fail on rocky linux 8[/+] on Oct 7, 2024AA-Turner commented on Oct 7, 2024 Are you able to test with pre releases, or attempt git bisect to help us narrow down the issue?
A JacobHenner commented on Oct 7, 2024
Are you able to test with pre releases, or attempt git bisect to help us narrow down the issue?
Yes - for the pre-releases, are there specific versions I should attempt? AA-Turner commented on Oct 7, 2024 I would attempt a manual bisection -- start with eg eg beta 1, then iterate through depending on if the failures are present. JacobHenner commented on Oct 7, 2024
I would attempt a manual bisection -- start with eg eg beta 1, then iterate through depending on if the failures are present.
I've reproduced the test_xml_etree error on all rc releases, and also a1. I'm testing some beta releases now.
Oddly, the test_hashlib error only occurs within our CI build environment, but not when building locally. The reason this is odd is because the same Dockerfile is used in both cases, with no known difference in build context that'd explain why it'd fail in one case and succeed in the other. I'll follow up on this one next. Reacted by Vladimir Kurilin JacobHenner commented on Oct 7, 2024 AA-Turner commented on Oct 8, 2024 Have you tried with a more recent version of expat (2.6+)? Do the tests succeed? JacobHenner commented on Oct 8, 2024
Have you tried with a more recent version of expat (2.6+)? Do the tests succeed?
Just did - test_xml_etree* succeeds with expat 2.6.3 built from source on Rocky 8, with all else remaining equal. vkurilin commented on Oct 8, 2024
Oddly, the test_hashlib error only occurs within our CI build environment, but not when building locally.
+1 to @JacobHenner – I have the exact same issue with test_hashlib. For 3.13.0 in CI, tests are failing both on rhel and debian. However, for all versions from 3.8 to 3.12, the tests pass without errors.
Locally, the Dockerfile builds without errors. JacobHenner commented on Oct 8, 2024
Oddly, the test_hashlib error only occurs within our CI build environment, but not when building locally.
+1 to @JacobHenner – I have the exact same issue with test_hashlib. For 3.13.0 in CI, tests are failing both on rhel and debian. However, for all versions from 3.8 to 3.12, the tests pass without errors.
Locally, the Dockerfile builds without errors.
What are you using to build your containers? Docker, podman, Kaniko? edmorley commented on Dec 6, 2024 Thank you for the replies.
However, there's been a bit of a misunderstanding. By "distro maintainer" I was referring to my not being a "maintainer of a distribution of the Linux OS", and not anything about being a "distributor of Python binaries".
Ideally I wouldn't have to be building Python binaries at all, however:
Perhaps things will improve in the future, given recent news about python-build-standalone, but for now we're stuck with building Python binaries.
Given our use-case, it seems likely we'll pick:
a) use the CPython bundled Expat (and be vulnerable until that bundle is updated and your CPython is updated)
...especially given that this is the configuration that CPython defaults to, and tests in its own CI. Reacted by Tianon Gravi Reacted by Tianon Gravi serhiy-storchaka commented on Dec 6, 2024 Do you also distribute Python tests? If yes, then you should distribute modified tests which ignore such failure (on selected platforms or always). If no, then you should not worry at all. edmorley commented on Dec 6, 2024
Do you also distribute Python tests? If yes, then you should distribute modified tests which ignore such failure (on selected platforms or always).
No we don't distribute tests to reduce resultant container image size (we build with --disable-test-modules).
If no, then you should not worry at all.
The failing tests with Ubuntu expat are still something that affects us, since:
- We need PGO, and PGO now requires the tests to pass (quite reasonably so!)
- If we patch the tests so that the failing tests are skipped, then it could just be masking a current or future expat incompatibility, per:
(More significantly) Is there an actual functionality regression when using xml.parsers.expat with Ubuntu 22.04's expat (that has partial backports), or is this a false-positive-test-only issue? Plus even if it's only a false positive now, how will we know there won't be a real regression in the future?
(from #125067 (comment)) hartwork commented on Dec 6, 2024
Do you also distribute Python tests? [..]. If no, then you should not worry at all.
I believe this picture is missing that CPython compiled against backported system Expat will use less of the recent Expat API internally (because the CPython build system assumes that API to be missing based on its version, which is legit) and so then even outside of tests pull requests like #115623 may not work as expected any more because system Expat is behaving differently than what CPython is rightfully expecting from the given Expat version number. So if I'm not missing something it would need patching where binaries are built against backported Expat, and not just the tests but also C code (or even just C code). A cheap trick that may work is to fake XML_COMBINED_VERSION to a different value but it only works when the given backport did not omit any behavior-changing patches in between. It will need close evaluation (and then patching) in any case. Compared to the pile of patches to CPython that other distributors need to apply, one patch is not too bad for a distribution of CPython with regard to cost. On a personal note, I think it's interesting that 99% of the time backporting appears to work just fine and then that other 1% shows how flawed the backporting approach is by design and that most of the time it just doesn't show.
CC @serhiy-storchaka @edmorley rgajason commented on Jul 7, 2025 FWIW, this issue exists on RHEL9/Rocky9 as well which ships expat 2.5.0. I've tested builds on Python 3.13.0-3.13.5 thus far. clemlesne commented on Jul 15, 2025 Also exists on Azure Linux 3 with Python 3.13.5. serhiy-storchaka commented on Jul 15, 2025 I do not think that this can be fixed on our side. Reacted by Sebastian Pipping whybeyoung commented on Oct 13, 2025 edmorley commented on Oct 13, 2025 @whybeyoung We (like docker-library/python) have had to give up on using --with-system-expat and instead use the libexpat version bundled with CPython as the least-worst alternative. (And given it matches the CPython configure defaults and what's tested in CPython's own CI, it feels fine as a long term solution to me.) Reacted by Tianon Gravi and Anthon van der Neut Reacted by Tianon Gravi StanFromIreland commented on Jun 9, 2026 Closing, as Serhiy noted above this isn't something we can fix, and there are workarounds.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|