GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-119169: Simplify `os.walk()` exception handling by barneygale · Pull Request #121435 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Jul 6, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Handle errors from os.scandir() and ScandirIterator similarly, which lets us loop over directory entries with for.

$ ./python -m timeit -s 'import os' 'list(os.walk("."))'
5 loops, best of 5: 41.5 msec per loop
5 loops, best of 5: 40.1 msec per loop
# --> 3.5% faster
  • Issue: Improve performance of os.walk() #119169

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@barneygale
pythonGH-119186: Slightly speed up os.walk()
abce81c
Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`.
@barneygale barneygale added the performance Performance or resource usage label Jul 6, 2024
@bedevere-app bedevere-app Bot added the awaiting core review label Jul 6, 2024
@barneygale barneygale changed the title GH-119186: Simplify os.walk() exception handling GH-119169: Simplify os.walk() exception handling Jul 6, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jul 6, 2024
Improve performance of os.walk() #119169
Open
serhiy-storchaka
serhiy-storchaka approved these changes Jul 8, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

This is not so much optimization as simplification.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

barneygale reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jul 8, 2024
@barneygale

barneygale commented Jul 8, 2024

Copy link
Copy Markdown
Contributor Author

Thanks Serhiy

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
barneygale merged commit db00fee into python:main Jul 8, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jul 8, 2024
@bedevere-bot

bedevere-bot commented Jul 8, 2024

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 LTO + PGO 3.x has failed when building commit db00fee.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/568/builds/6911) and take a look at the build logs.
  4. Check if the failure is related to this commit (db00fee) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/568/builds/6911

Failed tests:

  • test_smtplib

Failed subtests:

  • test_with_statement_QUIT_failure - test.test_smtplib.SMTPSimTests.test_with_statement_QUIT_failure
  • test_with_statement - test.test_smtplib.SMTPSimTests.test_with_statement

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/test/test_smtplib.py", line 1243, in test_with_statement_QUIT_failure
    smtp.noop()
    ~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 523, in noop
    return self.docmd("noop")
           ~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 430, in docmd
    return self.getreply()
           ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 403, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/test/test_smtplib.py", line 1233, in test_with_statement
    code, message = smtp.noop()
                    ~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 523, in noop
    return self.docmd("noop")
           ~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 430, in docmd
    return self.getreply()
           ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/smtplib.py", line 403, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Jul 8, 2024

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit db00fee.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/8522) and take a look at the build logs.
  4. Check if the failure is related to this commit (db00fee) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/8522

Failed tests:

  • test_smtplib

Failed subtests:

  • test_with_statement_QUIT_failure - test.test_smtplib.SMTPSimTests.test_with_statement_QUIT_failure
  • test_with_statement - test.test_smtplib.SMTPSimTests.test_with_statement

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_smtplib.py", line 1243, in test_with_statement_QUIT_failure
    smtp.noop()
    ~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 523, in noop
    return self.docmd("noop")
           ~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 430, in docmd
    return self.getreply()
           ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 403, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/test/test_smtplib.py", line 1233, in test_with_statement
    code, message = smtp.noop()
                    ~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 523, in noop
    return self.docmd("noop")
           ~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 430, in docmd
    return self.getreply()
           ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes-except-blake2/build/Lib/smtplib.py", line 403, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
@barneygale @estyxx
pythonGH-119169: Simplify os.walk() exception handling (python#121435)
7a70032
Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@barneygale @bedevere-bot @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.