GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-115133: test_xml_etree.py: Fix for Expat >=2.6.0 with reparse deferral (fixes #115133) by hartwork · Pull Request #115138 · python/cpython

Conversation

@hartwork

@hartwork hartwork commented Feb 7, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Fixes #115133

  • Issue: test.test_xml_etree*.XMLPullParserTest.test_simple_xml fails with (system) expat 2.6.0 #115133

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
hardfalcon reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-app bedevere-app Bot added awaiting review tests Tests in the Lib/test dir labels Feb 7, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Feb 7, 2024
test.test_xml_etree*.XMLPullParserTest.test_simple_xml fails with (system) expat 2.6.0 #115133
Closed
@hartwork
test_xml_etree.py: Fix for Expat >=2.6.0 with reparse deferral
a147a3e
@hartwork
hartwork force-pushed the issue-115133-fix-etree-xml-pull-parser-tests-for-expat-2-6-0 branch from 3c711d7 to a147a3e Compare February 7, 2024 14:50
@serhiy-storchaka

serhiy-storchaka commented Feb 7, 2024

Copy link
Copy Markdown
Member

It rather defeats the purpose of the test.

The purpose of the incremental parsing is that you can process the XML file without reading all data and building the full tree. This test tests that you get the parsed data as fast as the corresponding input data is available. The modified test only tests that it is available after feeding all input data and closing the parser. But you do not need the incremental parsing for this.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mgorny

mgorny commented Feb 7, 2024

Copy link
Copy Markdown
Contributor

I wonder how this affects stuff like XMPP where you actually need incremental parsing for things to work at all.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@Snild-Sony

Snild-Sony commented Feb 7, 2024

Copy link
Copy Markdown

This test tests that you get the parsed data as fast as the corresponding input data is available. The modified test only tests that it is available after feeding all input data and closing the parser. But you do not need the incremental parsing for this.

I'm not sure how to guarantee that one gets the data "as fast as the input", but it should at least be possible to assert that some parsing happens before close(). Would that be close enough?

I wonder how this affects stuff like XMPP where you actually need incremental parsing for things to work at all.

Probably not well, at least in general. With XMPP specifically, you may get lucky and always send a whole packet for parsing at once (or just generally have small tokens), in which case there will be no incomplete tokens and therefore no need for reparse deferral. But it will be very implementation-dependent.

For such a special usecase, this should work as a "flush" operation:

XML_SetReparseDeferralEnabled(parser, XML_FALSE);
XML_Parse(parser, NULL, 0, XML_FALSE); // or the last data of your "batch", if you know that ahead of time
XML_SetReparseDeferralEnabled(parser, XML_TRUE);
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@hardfalcon

hardfalcon commented Feb 7, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

In case it's relevant to anyone: Python 3.11.8, built with the patch from this pull request against expat 2.6.0, and also using expat 2.6.0 after installing said Python package, is working just fine for me, both for running Gajim (an XMPP client written in Python) and for running a matrix-synapse server.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@mweinelt mweinelt mentioned this pull request Feb 10, 2024
expat: 2.5.0 -> 2.6.0 NixOS/nixpkgs#287040
Merged
2 of 13 tasks
@serhiy-storchaka

serhiy-storchaka commented Feb 11, 2024

Copy link
Copy Markdown
Member

Thank you for your PR, but it was fixed by #115164 which keeps some of more strict testing.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka serhiy-storchaka closed this Feb 11, 2024
@hartwork hartwork mentioned this pull request Feb 13, 2024
Please expose Expat >=2.6.0 API function XML_SetReparseDeferralEnabled #115398
Closed
@hartwork
hartwork deleted the issue-115133-fix-etree-xml-pull-parser-tests-for-expat-2-6-0 branch February 17, 2024 13:54
@hartwork hartwork mentioned this pull request Feb 18, 2024
gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) #115623
Merged
gpshead pushed a commit that referenced this pull request Feb 29, 2024
@hartwork
gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (…
6a95676 
…GH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 2, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
ac019e3 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
This was referenced Mar 2, 2024
[3.12] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116248
Merged
[3.11] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116268
Merged
[3.10] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116270
Merged
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
0444649 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
2ab9c9b 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
@hartwork hartwork mentioned this pull request Mar 3, 2024
[3.9] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116272
Merged
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
1110d3e 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
@hartwork hartwork mentioned this pull request Mar 3, 2024
[3.8] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116275
Merged
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 3, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
fb7ea6f 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024
@hartwork @woodruffw
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
bd08a42 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
gpshead added a commit that referenced this pull request Mar 6, 2024
@hartwork @gpshead
[3.12] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023…
0a01ed6 
…-52425) (GH-115623) (GH-116248)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676)
(cherry picked from commit 73807eb)
(cherry picked from commit eda2963)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>
hartwork added a commit to hartwork/cpython that referenced this pull request Mar 6, 2024
@hartwork
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
3614f0b 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
github-actions Bot pushed a commit to m-aciek/python-docs-weblate that referenced this pull request Mar 6, 2024
[3.12] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023…
16e2972 
…-52425) (GH-115623) (GH-116248)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python/cpython#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676bb526261434dd068d6c49927c44d24a9b)
(cherry picked from commit 73807eb634315f70a464a18feaae33d9e065de09)
(cherry picked from commit eda2963378a3c292cf6bb202bb00e94e46ee6d90)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>

CPython-sync-commit-latest: 0a01ed6c2a116bd3e174fce33c21d84d650de569
gpshead added a commit that referenced this pull request Mar 6, 2024
@hartwork @gpshead
[3.11] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023…
fc9da96 
…-52425) (GH-115623) (#116268)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

(cherry picked from commit 6a95676)
(cherry picked from commit 73807eb)
(cherry picked from commit eda2963)

---------

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
@hartwork @gpshead
[3.10] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023…
516a6d4 
…-52425) (GH-115623) (GH-116270)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
@hartwork @gpshead
[3.9] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
2007624 
…52425) (GH-115623) (GH-116272)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
ambv pushed a commit that referenced this pull request Mar 6, 2024
@hartwork @gpshead
[3.8] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
854f645 
…52425) (GH-115623) (GH-116275)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from #115138 (comment) .

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

Co-authored-by: Gregory P. Smith <[email protected]>
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
@hartwork @adorilson
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
378a91f 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@hartwork @diegorusso
pythongh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-…
1069c2d 
…52425) (pythonGH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python#115138 (comment) .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.
jessecomeau87 pushed a commit to jessecomeau87/Python that referenced this pull request May 20, 2024
@hartwork
gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425)…
05e809a 
… (GH-115623)

Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from python/cpython#115138 (comment) .

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <[email protected]>
and by core dev Serhiy Storchaka.

(cherry picked from commit 6a95676)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test.test_xml_etree*.XMLPullParserTest.test_simple_xml fails with (system) expat 2.6.0

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@hartwork @serhiy-storchaka @mgorny @Snild-Sony @hardfalcon
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.