Description

colesbury

opened on Jan 30, 2023

Issue body actions

The PR #31696 attempts to fix the "leak" of file descriptors when the iterator is not exhausted. That PR fixes the warning, but not the underlying issue that the files aren't closed until the next tracing garbage collection cycle.

Note that there isn't truly a leak of file descriptors. The file descriptors are eventually closed when the file object is finalized (at cyclic garbage collection). The point of the ResourceWarning (in my understanding) is that waiting until the next garbage collection cycle means that you may temporarily have a lot of unwanted open file descriptors, which could exhaust the global limit or prevent successful writes to those files on Windows.

# run with ulimit -Sn 1000
import xml.etree.ElementTree as ET
import tempfile

import gc
gc.disable()

def run():
    with tempfile.NamedTemporaryFile("w") as f:
        f.write("<document />junk")

        for i in range(10000):
            it = ET.iterparse(f.name)
            del it

run()

On my system, after lowering the file descriptor limit to 1000 (via ulimit -Sn 1000) I get:

OSError: [Errno 24] Too many open files: '/tmp/tmpwwmd9gp6'

Linked PRs

• gh-101438: Avoid reference cycle in ElementTree.iterparse. #114269
• [3.12] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269) #114499
• [3.11] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269) #114500

Activity

1. colesbury

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jan 30, 2023
2. ericvsmith commented on Jan 30, 2023

   

   ericvsmith

   on Jan 30, 2023

   Member

   More actions

   I think this would require something like PEP 533 “Deterministic cleanup for iterators“ to solve.
3. colesbury commented on Jan 30, 2023

   

   colesbury

   on Jan 30, 2023

   ContributorAuthor

   More actions

   In this case, I think you can fix it just by avoiding reference cycles that involve the generator. Below is an example refactoring of iterparse.

   def iterparse(source, events=None, parser=None):
       # Use the internal, undocumented _parser argument for now; When the
       # parser argument of iterparse is removed, this can be killed.
       pullparser = XMLPullParser(events=events, _parser=parser)
       _root = None
   
       def iterator(source):
           nonlocal _root
           close_source = False
           try:
               if not hasattr(source, "read"):
                   source = open(source, "rb")
                   close_source = True
               yield None
               while True:
                   yield from pullparser.read_events()
                   # load event buffer
                   data = source.read(16 * 1024)
                   if not data:
                       break
                   pullparser.feed(data)
               root = pullparser._close_and_return_root()
               yield from pullparser.read_events()
               _root = root
           finally:
               if close_source:
                   source.close()
   
       class IterParseIterator(collections.abc.Iterator):
           def __init__(self, source):
               self.it = iterator(source)
           def __next__(self):
               return next(self.it)
           @property
           def root(self):
               nonlocal _root
               return _root
       it = IterParseIterator(source)
       del iterator, IterParseIterator
   
       next(it)
       return it

   This avoids two reference cycles that captured the generator created by iterator(source):

   1. Classes naturally form reference cycles, so move iterator(source) to the instance instead of a class attribute
   2. Move _root up so that iterator(source) doesn't refer to the instance it.

   I'm not sure about a few things including:

   • How do you test this?
   • Is it.root important? It's not documented. Is it only for testing?

   EDIT: the problem is not capturing of the file descriptor, but the generator iterator(source).
4. rhettinger commented on Jan 30, 2023

   

   rhettinger

   on Jan 30, 2023

   Contributor

   More actions

   @colesbury Doe the issue persist if you add gc.collect() after the del it?
5. colesbury commented on Jan 30, 2023

   

   colesbury

   on Jan 30, 2023

   ContributorAuthor

   More actions

   @rhettinger everything is cleaned up properly once the garbage collector runs. So adding a gc.collect() after the del it would collect the generator and close the file descriptor.
6. markshannon

   added a commit that references this issue on Apr 19, 2023

   pythongh-101438: possible fix for ElementTree.iterparse leak

   e3bd9af
7. iritkatriel

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 26, 2023
8. colesbury

   added a commit that references this issue on Jan 18, 2024

   pythongh-101438: Avoid reference cycle in ElementTree.iterparse.

   2119f17
9. serhiy-storchaka

   added a commit that references this issue on Jan 23, 2024

   gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269)

   ce01ab5
10. miss-islington

    added 2 commits that reference this issue on Jan 23, 2024

    pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

    e6d9756

    pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

    14e7028
11. serhiy-storchaka

    closed this as completedon Jan 23, 2024
12. serhiy-storchaka

    added

    3.11only security fixesonly security fixes

    3.12only security fixesonly security fixes

    3.13only security fixesonly security fixes

    on Jan 23, 2024
13. serhiy-storchaka

    added 2 commits that reference this issue on Jan 23, 2024

    [3.12] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…

    f63dec9

    [3.11] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…

    9344ede
14. aisk

    added a commit that references this issue on Feb 11, 2024

    pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

    c2034ab
15. Glyphack

    added a commit that references this issue on Sep 2, 2024

    pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

    11ff280

Issue actions