GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Change shutil.rmtree and os.walk to support very deep hierarchies · Issue #89727 · python/cpython

Change shutil.rmtree and os.walk to support very deep hierarchies #89727

New issue

Copy link

New issue

Copy link

Open

Change shutil.rmtree and os.walk to support very deep hierarchies#89727

Copy link

Labels

3.13only security fixesonly security fixesstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancementA feature request or enhancement

Description

AlexanderPatrakov

mannequin

opened

on Oct 21, 2021

Issue body actions

BPO	45564

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-21.22:21:02.146>
labels = ['library', '3.9', 'type-crash']
title = 'shutil.rmtree and os.walk are implemented using recursion, fail on deep hierarchies'
updated_at = <Date 2021-10-21.22:21:02.146>
user = 'https://bugs.python.org/AlexanderPatrakov'

bugs.python.org fields:

activity = <Date 2021-10-21.22:21:02.146>
actor = 'Alexander.Patrakov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-21.22:21:02.146>
creator = 'Alexander.Patrakov'
dependencies = []
files = []
hgrepos = []
issue_num = 45564
keywords = []
message_count = 1.0
messages = ['404687']
nosy_count = 1.0
nosy_names = ['Alexander.Patrakov']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45564'
versions = ['Python 3.9']

Linked PRs

Reacted by Stanislav Zmiev

Activity

AlexanderPatrakovmannequin

on Oct 21, 2021

MannequinAuthor

More actions

It is possible to create deep directory hierarchies that cannot be removed via shutil.rmtree or walked via os.walk, because these functions exceed the interpreter recursion limit. This may have security implications for web services (e.g. various webdisks) that have to clean up user-created mess or walk through it.

[aep@aep-haswell ~]$ mkdir /tmp/badstuff
[aep@aep-haswell ~]$ cd /tmp/badstuff
[aep@aep-haswell badstuff]$ for x in `seq 2048` ; do mkdir $x ; cd $x ; done
[aep@aep-haswell 103]$ cd
[aep@aep-haswell ~]$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.rmtree('/tmp/badstuff')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/shutil.py", line 726, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.9/shutil.py", line 663, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 992 more times]
  File "/usr/lib/python3.9/shutil.py", line 642, in _rmtree_safe_fd
    fullname = os.path.join(path, entry.name)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
    sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>> import os
>>> list(os.walk('/tmp/badstuff'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  File "/usr/lib/python3.9/os.py", line 418, in _walk
    yield from _walk(new_path, topdown, onerror, followlinks)
  [Previous line repeated 993 more times]
  File "/usr/lib/python3.9/os.py", line 412, in _walk
    new_path = join(top, dirname)
  File "/usr/lib/python3.9/posixpath.py", line 77, in join
    sep = _get_sep(a)
  File "/usr/lib/python3.9/posixpath.py", line 42, in _get_sep
    if isinstance(path, bytes):
RecursionError: maximum recursion depth exceeded while calling a Python object
>>>

AlexanderPatrakov
added
3.9 (EOL)end of lifeend of life

stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
on Oct 21, 2021
ezio-melotti
transferred this issue from on Apr 10, 2022
AlexWaygood
added
type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
and removed
type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
on Jul 10, 2022
jonburdo
added 2 commits that reference this issue on Nov 26, 2022
pythongh-89727: Fix os.walk RecursionError on deep trees
d106f5c
pythongh-89727: Fix os.walk RecursionError on deep trees
eeca07b
barneygale commented on Nov 27, 2022
barneygale
on Nov 27, 2022
Contributor
More actions
This also affects pathlib.Path.walk()
Reacted by Stanislav Zmiev
jonburdo
added a commit that references this issue on Nov 30, 2022
pythongh-89727: Fix os.walk RecursionError on deep trees
694c71e
jonburdo
added a commit that references this issue on Dec 15, 2022
pythongh-89727: Fix os.walk RecursionError on deep trees
99f4691
zmievsa
added a commit that references this issue on Dec 16, 2022
Merge branch 'main' into pythongh-89727/fix-pathlib.Path.walk-recursi…
7f36140
97 remaining items
miss-islington
added a commit that references this issue on May 30, 2024
pythonGH-89727: Fix FD leak on `os.fwalk()` generator finalization. (p…
ead49a2
barneygale
added 2 commits that reference this issue on May 30, 2024
[3.12] GH-89727: Fix FD leak on `os.fwalk()` generator finalization. (G…
d4a146d
[3.13] GH-89727: Fix FD leak on `os.fwalk()` generator finalization. (G…
083bf3a
barneygale commented on May 31, 2024
barneygale
on May 31, 2024
Contributor
More actions
One more for the pile: shutil.copytree() also calls itself recursively.
barneygale
added a commit that references this issue on Jun 1, 2024
GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (#119808)
53b1981
miss-islington
added a commit that references this issue on Jun 1, 2024
pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…
136d027
barneygale
added 5 commits that reference this issue on Jun 1, 2024
pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…
b900fdb
[3.12] pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep …
411a75d
[3.13] GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (G…
225c3cd
[3.12] GH-89727: Fix `shutil.rmtree()` recursion error on deep trees (G…
feb7870
pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…
39998dd
estyxx
added 4 commits that reference this issue on Jul 17, 2024
pythonGH-89727: Partially fix `shutil.rmtree()` recursion error on de…
459d4f3
pythonGH-89727: Fix `os.fwalk()` recursion error on deep trees (pytho…
6bb9185
pythonGH-89727: Fix FD leak on `os.fwalk()` generator finalization. (p…
984d064
pythonGH-89727: Fix `shutil.rmtree()` recursion error on deep trees (p…
24a1a10

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Change shutil.rmtree and os.walk to support very deep hierarchies #89727

Description

Linked PRs

Activity

AlexanderPatrakov commented on Oct 21, 2021

barneygale commented on Nov 27, 2022

97 remaining items

barneygale commented on May 31, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions