GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-89727: Improve os.walk complexity and speed by zmievsa · Pull Request #100671 · python/cpython

Conversation

@zmievsa

@zmievsa zmievsa commented Jan 1, 2023

Copy link
Copy Markdown
Contributor

Use isinstance for checking the top of the stack in os.walk to simplify it and speed it up.

  • Issue: Change shutil.rmtree and os.walk to support very deep hierarchies #89727

A small benchmark below. "." is this repository.
image

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@zmievsa
pythongh-89727: Improve os.walk complexity and speed
4127cf4
@bedevere-bot bedevere-bot added the awaiting review label Jan 1, 2023
@bedevere-bot bedevere-bot mentioned this pull request Jan 1, 2023
Change shutil.rmtree and os.walk to support very deep hierarchies #89727
Open
@blurb-it
📜🤖 Added by blurb_it.
5178f24
@AlexWaygood AlexWaygood added the performance Performance or resource usage label Jan 2, 2023
zmievsa added 2 commits January 2, 2023 04:24
@zmievsa
Remove unnecessary whitespace
c087697
@zmievsa
Merge branch 'pythongh-89727/minor-os-walk-refactoring' of github.com…
8164d36
…:ovsyanka83/cpython into pythongh-89727/minor-os-walk-refactoring
@zmievsa

zmievsa commented Jan 2, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@JelleZijlstra I can't assign a reviewer but you merged the prior PR related to os.walk so I decided that it makes sense to ping you here. I hope that's ok :))

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra
JelleZijlstra self-requested a review January 2, 2023 03:40
JelleZijlstra
JelleZijlstra reviewed Jan 2, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-01-01-23-57-00.gh-issue-89727.ojedHN.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra

JelleZijlstra commented Jan 2, 2023

Copy link
Copy Markdown
Member

Do you have a benchmark that shows this actually makes it faster? We lose a tuple allocation but gain a function call, so it's not completely obvious this should be faster.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Ovsyanka83 @JelleZijlstra
Update Misc/NEWS.d/next/Library/2023-01-01-23-57-00.gh-issue-89727.oj…
69cc847
…edHN.rst

Co-authored-by: Jelle Zijlstra <[email protected]>
@zmievsa

zmievsa commented Jan 2, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I made this MR because my initial benchmarking with pathlib.Path.walk and os.walk was quite obvious. However, after timing it a few more times I have gotten inconsistent results.

image

Seems like the isinstance version is just a tiny bit (insignificantly) faster but I would still argue that it makes more sense than the original because it is slightly simpler.

But what do you think?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

JelleZijlstra
JelleZijlstra approved these changes Jan 2, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels Jan 2, 2023
@JelleZijlstra

JelleZijlstra commented Jan 2, 2023

Copy link
Copy Markdown
Member

Thanks, seems like it's about a wash but since the new code is simpler and we never released the old version, seems fine to change this.

zmievsa reacted with hooray emoji zmievsa reacted with heart emoji
All reactions
  • 🎉 1 reaction
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra
JelleZijlstra merged commit 73097d9 into python:main Jan 2, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label Jan 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@zmievsa @JelleZijlstra @bedevere-bot @AlexWaygood @Ovsyanka83
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.