GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-151788: Speed up http.server directory listing by using os.scandir() by mjbommar · Pull Request #151789 · python/cpython

Conversation

@mjbommar

@mjbommar mjbommar commented Jun 20, 2026

Copy link
Copy Markdown
Contributor
  • Issue: Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir() #151788

SimpleHTTPRequestHandler.list_directory() called os.path.isdir() (a stat) and os.path.islink() (an lstat) for every entry — two stat-family syscalls per file. That is wasted work on any filesystem and dominates listing time for large directories; on network filesystems such as NFS, where each call is a round-trip, it becomes severe.

This switches to os.scandir(), whose DirEntry objects report the entry type from the directory read itself (POSIX d_type / NFS READDIRPLUS), eliminating the per-entry stats in the common case. CPython already made this exact migration for os.walk(), glob, and pathlib.Path.iterdir() (gh-117727); http.server was simply missed.

Behavior preserved

DirEntry.is_dir() / is_symlink() match os.path.isdir / os.path.islink semantics — same follow-symlinks behavior and same return-False-on-error behavior — verified across real dirs/files, symlink-to-dir (still rendered with @ but linked with /), symlink-to-file, and broken symlinks. The existing Lib/test/test_httpservers.py suite passes unchanged (92/92), including the undecodable/unencodable filename cases that exercise surrogate-escaped names.

Benchmark

Directory with 1000 files + 1000 dirs:

value
stat-family syscalls (strace), old 4088
stat-family syscalls, new 88 (constant interpreter startup; per-entry loop ≈ 0)
local filesystem wall-clock ~10× faster
emulated NFS (per-stat latency injected) listing drops from seconds to ~2 ms

Caveat (no overselling): the large NFS figures assume the filesystem reports entry types in the directory read (d_type; local filesystems and NFS with READDIRPLUS, the Linux default). If a mount returns DT_UNKNOWN, os.scandir() falls back to one cached lstat per entry — still fewer calls than today, and never worse.


This PR was prepared with AI assistance (Claude Code). I reviewed the change and benchmarks and can explain it in my own words.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mjbommar @claude
pythongh-151788: Speed up http.server directory listing by using os.s…
0606d02
…candir

SimpleHTTPRequestHandler.list_directory() called os.path.isdir() and
os.path.islink() for every entry, issuing two stat-family syscalls per
file. This is wasted work on any filesystem and dominates listing time
for large directories; on network filesystems such as NFS, where each
call is a round-trip, it becomes severe.

Use os.scandir(), whose DirEntry objects report the type from the
directory read itself (d_type / READDIRPLUS), eliminating the per-entry
stats in the common case and never doing more work than before.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@bedevere-app bedevere-app Bot mentioned this pull request Jun 20, 2026
Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir() #151788
Open
@bedevere-app bedevere-app Bot added the awaiting review label Jun 20, 2026
picnixz
picnixz requested changes Jun 20, 2026
View reviewed changes
Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Library/2026-06-20-09-00-26.gh-issue-151788.Gr8IxK.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting review label Jun 20, 2026
@bedevere-app

bedevere-app Bot commented Jun 20, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting changes label Jun 20, 2026
mjbommar and others added 2 commits June 20, 2026 14:55
@mjbommar @claude
pythongh-151788: Address review — OSError fallback and tone down NEWS
9ba9f68
- Wrap entry.is_dir()/is_symlink() in try/except OSError, falling back to
  False to exactly mirror os.path.isdir()/islink() (matches os.walk()).
- Reword the NEWS entry to avoid speedup-multiplier claims; describe it as
  improving list_directory() on systems with slow stat calls.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@mjbommar @claude
pythongh-151788: Address review — drop perf comment, tame NEWS wording
57bc982
- Remove the performance rationale from the inline comment; keep only the
  note explaining the OSError fallback (correctness, not performance).
- Reword the NEWS entry to state the mechanism without magnitude claims.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@mjbommar

mjbommar commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting change review and removed awaiting changes labels Jun 21, 2026
@bedevere-app

bedevere-app Bot commented Jun 21, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app
bedevere-app Bot requested a review from picnixz June 21, 2026 10:50
@picnixz

picnixz commented Jun 21, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Please fix the CI first before asking for a rereview.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mjbommar @claude
pythongh-151788: Fix NEWS entry doc reference
1138200
Drop the :meth: cross-reference to list_directory, which is not a
documented method and fails the docs "new NEWS nits" check.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@mjbommar

mjbommar commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

@picnixz sorry, thought this was just test flake. The expected checks are all green now other than your review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2026-06-20-09-00-26.gh-issue-151788.Gr8IxK.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@mjbommar @claude
pythongh-151788: Apply review suggestions
d47f9fb
- Drop the local `name` variable in favor of entry.name.
- Reword the OSError comment per review.
- Move the "Append /" comment above the is_dir check.
- Reword the NEWS entry per review (fix call -> calls).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/http/server.py Outdated
Comment on lines +916 to +919
displayname = entry.name + "/"
linkname = entry.name + "/"
if is_symlink:
displayname = entry.name + "@"

@picnixz picnixz Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you avoid having an LLM doing those suggestions please?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mjbommar mjbommar Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, AFK today but can do that tonight. Emacs doesn't work so well on phone. You want PR clean without Claude?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I (personally) want usually is:

  • human interaction (I don't want to communicate with an agent, that's out the question. I don't care whether the text has been written by the AI as long as it's not a C/C without any human review before sending it but I don't want an automated conversation)
  • commits that are always reviewed by a human before even committing them; in general this means that the AI is giving you the rough idea or contents but you are responsible for cleaning it up, and addressing things that could help the next review. In this case, I considered the name = entry.name to be un-necessary so reducing the diff only required to keep the same code and inlining that access. It didn't mean to rewrite all the occurrences with entry.name.

Our policy is that AI is a tool that can be used but it's not something meant to generate PRs for users that would later take credit for (I'm not saying you are, I'm saying that just how AI PRs usually go: most of the time, people just want to have a contribution and so they simply leave their agent in automode).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, the problem generally is not the AI, it's more that whenever there is an LLM, that LLM tends to choose things that are either useless, weird, and that would have been avoided if a human had directly written the lines. If you prefer writing your code through Claude, it's ok, but keep in mind that reviewers are not responsible for guiding the AI, it's the contributor. If we keep telling their agent through our review "no, not like that", it's as if we were directly using an agent in the least efficient way, and hence we are both losing our time.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mjbommar mjbommar Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. FWIW, Claude pointed out that your suggestion resulted in a NameError at string concat below, so it proposed this as easiest way to avoid name global and I accepted it

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, my bad actually it should have been displayname = linkname = name = entry.name. But yeah, in this case, I would have preferred to know why it decided to do so. And then I would have suggested:

displayname = linkname = name = entry.name

instead of the two lines. That's why I prefer that you tell me this first before changing otherwise I would get confused.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Jun 21, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to reiterate: there is nothing wrong with using Claude (the only think I don't like about it is the long summaries in every PR and trivial things it reports like "all tests pass" which is something I would expect for a PR otherwise it doesn't make sense). It's just that having a 3rd actor usually creates more back and forth between the reviewer and the reviewee.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz
picnixz reviewed Jun 21, 2026
View reviewed changes
Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/http/server.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

mjbommar and others added 2 commits June 23, 2026 08:02
@mjbommar @picnixz
Apply suggested rewrite for name
b1e3fd4
Co-authored-by: Bénédikt Tran <[email protected]>
@mjbommar
pythongh-151788: Normalize line endings to LF in list_directory
09f2b44
The applied suggestion introduced CRLF on the edited lines; normalize
them back to LF so the mixed-line-ending check passes.
@mjbommar

mjbommar commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app

bedevere-app Bot commented Jun 23, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app
bedevere-app Bot requested a review from picnixz June 23, 2026 14:02
picnixz
picnixz approved these changes Jun 23, 2026
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting change review labels Jun 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 90 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting merge stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@mjbommar @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.