GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-126033: fix a crash in `xml.etree.ElementTree.Element.remove` when concurrent mutations happen by picnixz · Pull Request #126124 · python/cpython

Conversation

@picnixz

@picnixz picnixz commented Oct 29, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

No need for versioning. Holding a strong reference on the child being compared is enough if we additionally check that the number of children is not changed after the comparison.

The test coverage could be refined but I don't think I need to overcomplicate it even more. I still think we can have a UAF because when one clears the list in __eq__ but I couldn't find a way to trigger it so I just increfed the item before calling __eq__. If anyone thinks it's not needed, please tell me why.

The patch for find* may be more complicated, which is why I splitted the task into three (one for introducing the versioning so that we can work on those failures in parallel).

  • Issue: UAF: xml.etree.ElementTree.Element.remove when concurrent mutations happen #126033

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz added 5 commits October 27, 2024 15:32
@picnixz
Add versioning to XML elements.
abc121c
@picnixz
fix tests
4efa517
@picnixz
fix portability issues
4ca3cf0
@picnixz
fixup
a1950d1
@picnixz
unify versioning
9b6f559
@bedevere-app bedevere-app Bot added the awaiting review label Oct 29, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Oct 29, 2024
UAF: xml.etree.ElementTree.Element.remove when concurrent mutations happen #126033
Closed
@picnixz
picnixz requested a review from sobolevn October 29, 2024 12:31
picnixz added 2 commits October 29, 2024 13:33
@picnixz
handle evil mutations in Element.remove
00a7a7e
@picnixz
blurb
59ade8f
@picnixz
picnixz force-pushed the fix/xml-evil-remove-126033 branch from bb4f6d2 to 59ade8f Compare October 29, 2024 12:33
sobolevn
sobolevn reviewed Oct 29, 2024
View reviewed changes
Comment thread Modules/_elementtree.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz requested a review from serhiy-storchaka December 6, 2024 16:55
@picnixz picnixz added the topic-XML label Dec 6, 2024
serhiy-storchaka
serhiy-storchaka reviewed Dec 6, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is adding version necessary? Would not keeping a strong reference to a child be enough? Look at the list.remove() code.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Misc/NEWS.d/next/Library/2024-10-29-12-59-45.gh-issue-126033.sM3uCn.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_xml_etree.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Dec 6, 2024

Copy link
Copy Markdown
Member Author

Would not keeping a strong reference to a child be enough? Look at the list.remove() code.

Oh yes. I'll try using this one as a basis.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Dec 6, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Would not keeping a strong reference to a child be enough? Look at the list.remove() code.

Ah I know why I couldn't make it work! it's because the children is not a list itself, but just a huge memory area that was dynamically allocated using PyMem_Malloc. What do you suggest I do?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
improve NEWS entry formulation
7fc9932
@serhiy-storchaka

serhiy-storchaka commented Dec 6, 2024

Copy link
Copy Markdown
Member

The list storage is also just a huge memory area that was dynamically allocated using PyMem_Malloc. I do not see difference.

At worst, you can get different exceptions (or no exception) in C and Python implementations, but this is fine.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Dec 6, 2024

Copy link
Copy Markdown
Member Author

The list storage is also just a huge memory area that was dynamically allocated using PyMem_Malloc. I do not see difference.

The difference is that the list object uses Py_SIZE but it's not possible to use this on the current implementation of element trees (or is there?)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Dec 6, 2024

Copy link
Copy Markdown
Member

Py_SIZE or self->extra->length -- there is not much difference between them. Both just read a word from memory. You may need to check that self->extra was not set to NULL during iteration, this is the only difference.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Dec 6, 2024

Copy link
Copy Markdown
Member Author

Py_SIZE or self->extra->length -- there is not much difference between them

I'm pretty sure I first tried using INCREF/DECREF as usual and encountered an issue with that but I can't remember how. I'll try reproducing this tomorrow.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz marked this pull request as draft December 6, 2024 18:03
@bedevere-app bedevere-app Bot removed the awaiting review label Dec 6, 2024
picnixz added 2 commits December 8, 2024 14:06
@picnixz
remove versioning
e30756d
@picnixz
fix tests
f2b5bb1
@picnixz
picnixz marked this pull request as ready for review December 8, 2024 13:08
@bedevere-app bedevere-app Bot added the awaiting review label Dec 8, 2024
@picnixz
picnixz requested review from serhiy-storchaka and sobolevn December 8, 2024 13:50
picnixz and others added 5 commits February 4, 2025 18:16
@picnixz @vstinner
Update Misc/NEWS.d/next/Library/2024-10-29-12-59-45.gh-issue-126033.s…
bc52c04
…M3uCn.rst

Co-authored-by: Victor Stinner <[email protected]>
@picnixz
remove un-necessary subtest parameter
f4a4dae
@picnixz
Reduce the visual size of the tests
09a9fa9
@picnixz
fixup docs
f5d352f
@picnixz
Merge branch 'main' into fix/xml-evil-remove-126033
2b47468
@picnixz
picnixz requested a review from vstinner February 12, 2025 11:21
picnixz added 2 commits February 22, 2025 11:33
@picnixz
Merge branch 'main' into fix/xml-evil-remove-126033
dc576ab
@picnixz
Merge branch 'main' into fix/xml-evil-remove-126033
ce66ac7
@picnixz

picnixz commented Mar 14, 2025

Copy link
Copy Markdown
Member Author

@vstinner @serhiy-storchaka friendly ping in case you forgot about this one

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Mar 14, 2025

Copy link
Copy Markdown
Member

As I wrote, I don't think that 200 lines of tests are needed. I would prefer way less tests.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Mar 14, 2025

Copy link
Copy Markdown
Member Author

I can remove the special test case which makes the test explode as it's only the Python implementation if you want. It would reduce the test by half at least (most of the line length is taken by comments that are needed to explain the discrepency between the Python and the C implementation).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
do not pedantically test the pure Python implementation
04cb600
@picnixz

picnixz commented Mar 15, 2025

Copy link
Copy Markdown
Member Author

@vstinner I've removed the pedantic check. However, we still need to have that much of tests because del root[:] and root.clear() behave differently although they should do the same. The reproducer I had did not make crash del root[:], but it made crash root.clear().

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner approved these changes Mar 31, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The C code change LGTM, I didn't review the tests.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Mar 31, 2025
@picnixz
Merge branch 'main' into fix/xml-evil-remove-126033
a0c2324
@picnixz
picnixz merged commit bab1398 into python:main Mar 31, 2025
@miss-islington-app

miss-islington-app Bot commented Mar 31, 2025

Copy link
Copy Markdown

Thanks @picnixz for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
picnixz deleted the fix/xml-evil-remove-126033 branch March 31, 2025 10:31
@bedevere-app bedevere-app Bot removed the awaiting merge label Mar 31, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 31, 2025
@picnixz @miss-islington
pythongh-126033: fix UAF in xml.etree.ElementTree.Element.remove wh…
1c6193c
…en concurrent mutations happen (pythonGH-126124)

(cherry picked from commit bab1398)

Co-authored-by: Bénédikt Tran <[email protected]>
@bedevere-app

bedevere-app Bot commented Mar 31, 2025

Copy link
Copy Markdown

GH-131929 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Mar 31, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 31, 2025
@picnixz @miss-islington
pythongh-126033: fix UAF in xml.etree.ElementTree.Element.remove wh…
e1aa52c
…en concurrent mutations happen (pythonGH-126124)

(cherry picked from commit bab1398)

Co-authored-by: Bénédikt Tran <[email protected]>
@bedevere-app

bedevere-app Bot commented Mar 31, 2025

Copy link
Copy Markdown

GH-131930 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Mar 31, 2025
picnixz added a commit that referenced this pull request Mar 31, 2025
@miss-islington @picnixz
[3.13] gh-126033: fix UAF in xml.etree.ElementTree.Element.remove w…
b41c8cc
…hen concurrent mutations happen (GH-126124) (#131929)

gh-126033: fix UAF in `xml.etree.ElementTree.Element.remove` when concurrent mutations happen (GH-126124)
(cherry picked from commit bab1398)

Co-authored-by: Bénédikt Tran <[email protected]>
picnixz added a commit that referenced this pull request Mar 31, 2025
@miss-islington @picnixz
[3.12] gh-126033: fix UAF in xml.etree.ElementTree.Element.remove w…
5d4e891
…hen concurrent mutations happen (GH-126124) (#131930)

gh-126033: fix UAF in `xml.etree.ElementTree.Element.remove` when concurrent mutations happen (GH-126124)
(cherry picked from commit bab1398)

Co-authored-by: Bénédikt Tran <[email protected]>
@picnixz picnixz mentioned this pull request Mar 31, 2025
Update Element.remove exception message when an element is not found #131938
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@sobolevn sobolevn Awaiting requested review from sobolevn
@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-XML

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@picnixz @serhiy-storchaka @vstinner @sobolevn
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.