GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Improve performance of `os.walk()` · Issue #119169 · python/cpython

Improve performance of os.walk() #119169

New issue
Copy link
New issue
Copy link
Open
Open
Improve performance of os.walk()#119169
Copy link
Labels
performancePerformance or resource usagePerformance or resource usage

Description

@barneygale
barneygale
opened on May 19, 2024
Issue body actions

There are a couple of minor performance improvements possible in os.walk():

  • We don't need to manually pump the os.scandir iterator, given we handle exceptions from next() like exceptions from scandir() itself, i.e. by ignoring the problematic directory and moving on. We can use a for loop like filthy casuals.
  • In bottom-up mode, we can handle exceptions from entry.is_symlink() in the same block as those from entry.is_dir(), which avoids a few temporary variables.
  • In top-down mode, we can call os.path.join() once on a parent directory rather than for each child path.

Linked PRs

  • GH-119169: Speed up os.walk(topdown=False) #119186
  • GH-119169: Skip reversing sibling directories in os.[f]walk(topdown=False) #119473
  • GH-119169: Implement pathlib.Path.walk() using os.walk() #119573
  • [3.13] GH-119169: Implement pathlib.Path.walk() using os.walk() (GH-119573) #119750
  • GH-119169: Slightly speed up os.walk(topdown=True) #121431
  • GH-119169: Simplify os.fwalk() exception handling #121432
  • GH-119169: Speed up os.fwalk(topdown=False) #121433
  • GH-119169: Simplify os.walk() exception handling #121435
Reacted by Nice Zombies, Wulian233 and lit

Activity

  1. barneygale
    added
    performancePerformance or resource usagePerformance or resource usage
    on May 19, 2024
  2. barneygale commented on May 19, 2024

    @barneygale
    barneygale
    on May 19, 2024
    ContributorAuthor
    More actions

    In bottom-up mode, we can handle exceptions from entry.is_symlink() in the same block as those from entry.is_dir(), which avoids a few temporary variables.

    Upon closer inspection, I think I was wrong. Symlinks to directories where is_symlink() raises should still be categorised as dirs.

  3. barneygale
    added 5 commits that reference this issue on May 19, 2024

    pythonGH-119169: Speed up `os.walk()`

    74a495c

    pythonGH-119169: Skip reversing sibling directories in `os.walk(topdo…

    469a9cf

    pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()`

    fedc8a5

    pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()`

    1f7d5ad

    GH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (#119573)

    7ff61f5
  4. miss-islington
    added a commit that references this issue on May 29, 2024

    pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (p…

    3668c1a
  5. barneygale
    added a commit that references this issue on May 29, 2024

    [3.13] GH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (G…

    a7aa7c4
  6. barneygale
    added 2 commits that reference this issue on Jul 6, 2024

    pythonGH-119169: Slightly speed up `os.fwalk()`

    d18bfb9

    pythonGH-119169: Speed up `os.fwalk(topdown=False)`

    bedf76e
  7. serhiy-storchaka commented on Jul 8, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Jul 8, 2024
    Member
    More actions

    There are too many open PRs. Are they independent? Alternative? What should I review?

  8. barneygale
    added a commit that references this issue on Jul 8, 2024

    GH-119169: Simplify `os.walk()` exception handling (#121435)

    db00fee
  9. barneygale commented on Jul 8, 2024

    @barneygale
    barneygale
    on Jul 8, 2024
    ContributorAuthor
    More actions

    There are too many open PRs. Are they independent? Alternative? What should I review?

    The two draft PRs (GH-121433 and GH-119186) depend on GH-119473. The three non-draft PRs are independent and can be reviewed/accepted/rejected in any order or combination.

  10. estyxx
    added 2 commits that reference this issue on Jul 17, 2024

    pythonGH-119169: Implement `pathlib.Path.walk()` using `os.walk()` (p…

    b64cd9f

    pythonGH-119169: Simplify `os.walk()` exception handling (python#121435)

    7a70032
  11. encukou
    added a commit that references this issue on Sep 3, 2025

    GH-119169: Slightly speed up `os.walk(topdown=True)` (GH-121431)

    b19ad11
  12. lkollar
    added a commit that references this issue on Sep 9, 2025

    pythonGH-119169: Slightly speed up `os.walk(topdown=True)` (pythonGH-…

    02b5cb3
  13. serhiy-storchaka
    added this to OS issues 💻on Jul 3, 2026
  14. github-project-automation
    moved this to Todo in OS issues 💻on Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagePerformance or resource usage

    Projects

    • OS issues 💻
      Status
      Todo
      Show more project fields

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.