GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-119169: Speed up `os.walk(topdown=False)` by barneygale · Pull Request #119186 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented May 19, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Add entries to the stack while iterating over os.scandir() results, rather than afterwards.

  • Issue: Improve performance of os.walk() #119169

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@barneygale
pythonGH-119169: Speed up os.walk()
74a495c 
Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`. In top-down mode, call
`os.path.join()` at most once per iteration.
@bedevere-app bedevere-app Bot added the awaiting core review label May 19, 2024
@bedevere-app bedevere-app Bot mentioned this pull request May 19, 2024
Improve performance of os.walk() #119169
Open
@barneygale barneygale added performance Performance or resource usage and removed awaiting core review labels May 19, 2024
@vstinner

vstinner commented May 20, 2024

Copy link
Copy Markdown
Member

cc @serhiy-storchaka
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
barneygale added 2 commits May 23, 2024 09:26
@barneygale
Merge branch 'main' into os-walk-speedup
eb4b6ff
@barneygale
Simplify bottom-up implementation.
46a3ba5
@barneygale
barneygale marked this pull request as draft May 23, 2024 08:46
@barneygale

barneygale commented May 23, 2024

Copy link
Copy Markdown
Contributor Author

#119473 fixes the test failure.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
barneygale added 2 commits May 30, 2024 05:06
@barneygale
Merge branch 'main' into os-walk-speedup
2d957a8
@barneygale
Apply the same changes to fwalk()
a638202
@barneygale barneygale changed the title GH-119169: Speed up os.walk() GH-119169: Speed up os.[f]walk() May 30, 2024
@barneygale
Simplify code slightly
91b5a78
@barneygale barneygale mentioned this pull request Jul 6, 2024
GH-119169: Skip reversing sibling directories in os.[f]walk(topdown=False) #119473
Closed
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 6, 2024
@barneygale
pythonGH-119186: Slightly speed up os.walk(topdown=True)
e5f62ec 
When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
barneygale added 5 commits July 6, 2024 14:39
@barneygale
Simplify exception handling in fwalk()
b32c1c0
@barneygale
Merge branch 'main' into os-walk-speedup
8c7cf04
@barneygale
Undo trailing slash optimisation (moved to another PR)
efd31e2
@barneygale
Undo fwalk() changes.
5b89c57
@barneygale
Fix news entry.
525618c
@barneygale barneygale changed the title GH-119169: Speed up os.[f]walk() GH-119169: Speed up os.walk() Jul 6, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 6, 2024
@barneygale
pythonGH-119186: Slightly speed up os.walk()
abce81c 
Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`.
@barneygale barneygale changed the title GH-119169: Speed up os.walk() GH-119169: Speed up os.walk(topdown=False) Jul 6, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 8, 2024
@barneygale
Merge branch 'main' into pythongh-119186-joining
6534a36
@barneygale
Merge branch 'main' into os-walk-speedup
d3af655
@barneygale

barneygale commented Jul 8, 2024

Copy link
Copy Markdown
Contributor Author

Closing because reversing sibling traversal order isn't kosher - see discussion on #119473.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale closed this Jul 8, 2024
encukou added a commit to barneygale/cpython that referenced this pull request Sep 2, 2025
@encukou
Merge branch 'main' into pythongh-119186-joining
7d6bd57
encukou pushed a commit that referenced this pull request Sep 3, 2025
@barneygale
GH-119169: Slightly speed up os.walk(topdown=True) (GH-121431)
b19ad11 
GH-119186: Slightly speed up `os.walk(topdown=True)`

When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
lkollar pushed a commit to lkollar/cpython that referenced this pull request Sep 9, 2025
@barneygale @lkollar
pythonGH-119169: Slightly speed up os.walk(topdown=True) (pythonGH-…
02b5cb3 
…121431)

pythonGH-119186: Slightly speed up `os.walk(topdown=True)`

When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@barneygale @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.