GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-101438: Avoid reference cycle in ElementTree.iterparse. by colesbury · Pull Request #114269 · python/cpython

Conversation

@colesbury

@colesbury colesbury commented Jan 18, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Refactor IterParseIterator to avoid a reference cycle between the iterator() function and the IterParseIterator() instance. This leads to more prompt clean-up of the "source" file if the returned iterator is not exhausted and not otherwise part of a reference cycle.

This also avoids a test failure in the GC implementation for the free-threaded build (#114262): if the "source" file is finalized before the iterator() generator, a ResourceWarning is issued leading to a failure in test_iterparse() in test_xml_etree.py. In theory, this warning can occur in the default build as well, but that is much less likely to occur because it would require an unlucky scheduling of the GC between creation of the generator and the file object in order to change the order of finalization.

  • Issue: ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@colesbury
pythongh-101438: Avoid reference cycle in ElementTree.iterparse.
2119f17 
Refactor IterParseIterator to avoid a reference cycle between the
iterator() function and the IterParseIterator() instance. This leads to
more prompt clean-up of the "source" file if the returned iterator is
not exhausted and not otherwise part of a reference cycle.

This also avoids a test failure in the GC implementation for the
free-threaded build: if the "source" file is finalized before the
"iterator()" generator, a ResourceWarning is issued leading to a
failure in test_iterparse(). In theory, this warning can occur in
the default build as well, but is much less likely because it would
require an unlucky scheduling of the GC between creation of the generator
and the file object in order to change the order of finalization.
@colesbury colesbury added topic-XML topic-free-threading labels Jan 18, 2024
@bedevere-app bedevere-app Bot added the awaiting review label Jan 18, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jan 18, 2024
ElementTree.iterparse "leaks" file descriptor when not exhausted #101438
Closed
@serhiy-storchaka

serhiy-storchaka commented Jan 19, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

A class with __next__() method adds an overhead. See #69824, it made benchmarks 2 times slower.

Please test how this change affects the results of the xml_etree_iterparse benchmark in https://github.com/python/pyperformance.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury
Avoid regression in bm_xml_etree performance.
8fabc7c 
This avoids the `__next__` wrapper and the `root` property, both of
which had a performance impact on the iterparse benchmark in
bm_xml_etree.
@colesbury

colesbury commented Jan 19, 2024

Copy link
Copy Markdown
Contributor Author

Thanks for the pointer @serhiy-storchaka. It had about a 15% regression. I rewrote it to avoid the regression. On my machine, I'm seeing 1.37-1.38 seconds for ten iterations of the iterparse benchmark (both before and after this PR).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury
colesbury requested a review from serhiy-storchaka January 19, 2024 19:46
@colesbury
Minor simplification
1ae917e
@colesbury

colesbury commented Jan 23, 2024

Copy link
Copy Markdown
Contributor Author

@serhiy-storchaka, would you be able to review this?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
serhiy-storchaka
serhiy-storchaka reviewed Jan 23, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a test?

There is an existing test

        # Not exhausting the iterator still closes the resource (bpo-43292)
        with warnings_helper.check_no_resource_warning(self):
            it = iterparse(TESTFN)
            del it

It was purposed to test that source.close() is called, but it seems that it was not called at all.

Perhaps running it in a loop can test this issue. And this code can be moved into a separate method.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/xml/etree/ElementTree.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@colesbury @serhiy-storchaka
Update Lib/xml/etree/ElementTree.py
05baaad 
Co-authored-by: Serhiy Storchaka <[email protected]>
@colesbury

colesbury commented Jan 23, 2024

Copy link
Copy Markdown
Contributor Author

@serhiy-storchaka, I don't think we can do better than the current test. That test does catch the behavior. See https://github.com/python/cpython/actions/runs/7617715570/job/20747338297 for the test failure.
serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@colesbury

colesbury commented Jan 23, 2024

Copy link
Copy Markdown
Contributor Author

I don't think a loop will do much either. The order of finalization is not specified, but in practice it is pretty deterministic. A loop will make the test slower, but not much more reliable.
serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
serhiy-storchaka
serhiy-storchaka approved these changes Jan 23, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jan 23, 2024
@serhiy-storchaka serhiy-storchaka added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Jan 23, 2024
@serhiy-storchaka
Update 2024-01-18-22-29-28.gh-issue-101438.1-uUi_.rst
307b375
@serhiy-storchaka
serhiy-storchaka enabled auto-merge (squash) January 23, 2024 19:52
@serhiy-storchaka

serhiy-storchaka commented Jan 23, 2024

Copy link
Copy Markdown
Member

Thank you for your contribution @colesbury. If you look at the history of this code, you will see that it is a continuous fight against a reference loops. So I'm especially grateful for this fix.
colesbury reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka
serhiy-storchaka merged commit ce01ab5 into python:main Jan 23, 2024
@miss-islington-app

miss-islington-app Bot commented Jan 23, 2024

Copy link
Copy Markdown

Thanks @colesbury for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 23, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 23, 2024
@colesbury @miss-islington
pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…
e6d9756 
…honGH-114269)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
(cherry picked from commit ce01ab5)

Co-authored-by: Sam Gross <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 23, 2024
@colesbury @miss-islington
pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…
14e7028 
…honGH-114269)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
(cherry picked from commit ce01ab5)

Co-authored-by: Sam Gross <[email protected]>
@bedevere-app

bedevere-app Bot commented Jan 23, 2024

Copy link
Copy Markdown

GH-114499 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Jan 23, 2024
@bedevere-app

bedevere-app Bot commented Jan 23, 2024

Copy link
Copy Markdown

GH-114500 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.11 only security fixes label Jan 23, 2024
serhiy-storchaka pushed a commit that referenced this pull request Jan 23, 2024
@miss-islington @colesbury
[3.12] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…
f63dec9 
…-114269) (GH-114499)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
(cherry picked from commit ce01ab5)

Co-authored-by: Sam Gross <[email protected]>
serhiy-storchaka pushed a commit that referenced this pull request Jan 23, 2024
@miss-islington @colesbury
[3.11] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…
9344ede 
…-114269) (GH-114500)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
(cherry picked from commit ce01ab5)

Co-authored-by: Sam Gross <[email protected]>
@colesbury
colesbury deleted the gh-101438-iterparse branch January 24, 2024 17:28
Prometheus3375
Prometheus3375 reviewed Jan 28, 2024
View reviewed changes
Comment thread Lib/xml/etree/ElementTree.py
Comment on lines +1258 to +1260
it = IterParseIterator()
wr = weakref.ref(it)
del IterParseIterator

@Prometheus3375 Prometheus3375 Jan 28, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious, why previously both iterator and IterParseIterator names were deleted, but now only IterParseIterator? And what is the purpose of this statement in the first place? I was thinking that iterator.__closure__ stores references to these objects; therefore, unnecessary references should be deleted. However, as per my checks, closure stores only referenced variables inside; pullparser, close_source and wr in this case.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@Prometheus3375 Prometheus3375 Jan 28, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also noticed that it.root = None was deleted. This fact is not documented, but this may still cause unintended errors on the user side if they use root.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@colesbury colesbury Jan 28, 2024

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are right about the it.root = None. I did not intend a behavioral change here, so it seems like a good idea to add it back.

I don't think the del statements matter one way or the other. They look like they break a cycle, but not really, but they also are harmless.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Prometheus3375 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@Prometheus3375 Prometheus3375 mentioned this pull request Jan 29, 2024
Fix unintended behavior change in elementtree introduced in #114269 #114737
Closed
colesbury added a commit to colesbury/cpython that referenced this pull request Jan 30, 2024
@colesbury
pythongh-114737: Revert change to ElementTree.iterparse "root" attribute
212ef9f 
Prior to pythongh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
@colesbury colesbury mentioned this pull request Jan 30, 2024
gh-114737: Revert change to ElementTree.iterparse "root" attribute #114755
Merged
serhiy-storchaka pushed a commit that referenced this pull request Jan 31, 2024
@colesbury
gh-114737: Revert change to ElementTree.iterparse "root" attribute (G…
66f95ea 
…H-114755)

Prior to gh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 31, 2024
@colesbury @miss-islington
pythongh-114737: Revert change to ElementTree.iterparse "root" attrib…
8ebd614 
…ute (pythonGH-114755)

Prior to pythongh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
(cherry picked from commit 66f95ea)

Co-authored-by: Sam Gross <[email protected]>
@miss-islington miss-islington mentioned this pull request Jan 31, 2024
[3.12] gh-114737: Revert change to ElementTree.iterparse "root" attribute (GH-114755) #114798
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 31, 2024
@colesbury @miss-islington
pythongh-114737: Revert change to ElementTree.iterparse "root" attrib…
e842682 
…ute (pythonGH-114755)

Prior to pythongh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
(cherry picked from commit 66f95ea)

Co-authored-by: Sam Gross <[email protected]>
@miss-islington miss-islington mentioned this pull request Jan 31, 2024
[3.11] gh-114737: Revert change to ElementTree.iterparse "root" attribute (GH-114755) #114799
Merged
serhiy-storchaka pushed a commit that referenced this pull request Jan 31, 2024
@miss-islington @colesbury
[3.12] gh-114737: Revert change to ElementTree.iterparse "root" attri…
5d7b90d 
…bute (GH-114755) (GH-114798)

Prior to gh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
(cherry picked from commit 66f95ea)

Co-authored-by: Sam Gross <[email protected]>
serhiy-storchaka pushed a commit that referenced this pull request Jan 31, 2024
@miss-islington @colesbury
[3.11] gh-114737: Revert change to ElementTree.iterparse "root" attri…
3982049 
…bute (GH-114755) (GH-114799)

Prior to gh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
(cherry picked from commit 66f95ea)

Co-authored-by: Sam Gross <[email protected]>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@colesbury @aisk
pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…
c2034ab 
…honGH-114269)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@colesbury @aisk
pythongh-114737: Revert change to ElementTree.iterparse "root" attrib…
d23850c 
…ute (pythonGH-114755)

Prior to pythongh-114269, the iterator returned by ElementTree.iterparse was
initialized with the root attribute as None. This restores the previous
behavior.
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@colesbury @Glyphack
pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…
11ff280 
…honGH-114269)

The iterator returned by ElementTree.iterparse() may hold on to a file
descriptor. The reference cycle prevented prompt clean-up of the file
descriptor if the returned iterator was not exhausted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
+1 more reviewer
@Prometheus3375 Prometheus3375 Prometheus3375 left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-free-threading topic-XML

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@colesbury @serhiy-storchaka @Prometheus3375
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.