GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-151788: Speed up http.server directory listing by using os.scandir() by mjbommar · Pull Request #151789 · python/cpython Conversation
Contributor
This switches to Behavior preserved
BenchmarkDirectory with 1000 files + 1000 dirs:
Caveat (no overselling): the large NFS figures assume the filesystem reports entry types in the directory read ( This PR was prepared with AI assistance (Claude Code). I reviewed the change and benchmarks and can explain it in my own words.
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions…candir SimpleHTTPRequestHandler.list_directory() called os.path.isdir() and os.path.islink() for every entry, issuing two stat-family syscalls per file. This is wasted work on any filesystem and dominates listing time for large directories; on network filesystems such as NFS, where each call is a round-trip, it becomes severe. Use os.scandir(), whose DirEntry objects report the type from the directory read itself (d_type / READDIRPLUS), eliminating the per-entry stats in the common case and never doing more work than before. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
picnixz
requested changes
Uh oh!
There was an error while loading. Please reload this page. Uh oh!
There was an error while loading. Please reload this page. Uh oh!
There was an error while loading. Please reload this page.
All reactions
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. - Wrap entry.is_dir()/is_symlink() in try/except OSError, falling back to False to exactly mirror os.path.isdir()/islink() (matches os.walk()). - Reword the NEWS entry to avoid speedup-multiplier claims; describe it as improving list_directory() on systems with slow stat calls. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> - Remove the performance rationale from the inline comment; keep only the note explaining the OSError fallback (correctness, not performance). - Reword the NEWS entry to state the mechanism without magnitude claims. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Contributor
Author
All reactions
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page.
All reactions
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page.
|
There was an error while loading. Please reload this page.
|
Please fix the CI first before asking for a rereview. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Drop the :meth: cross-reference to list_directory, which is not a documented method and fails the docs "new NEWS nits" check. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
@picnixz sorry, thought this was just test flake. The expected checks are all green now other than your review |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
- Drop the local `name` variable in favor of entry.name. - Reword the OSError comment per review. - Move the "Append /" comment above the is_dir check. - Reword the NEWS entry per review (fix call -> calls). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
| displayname = entry.name + "/" | ||
| linkname = entry.name + "/" | ||
| if is_symlink: | ||
| displayname = entry.name + "@" |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Can you avoid having an LLM doing those suggestions please?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Sure, AFK today but can do that tonight. Emacs doesn't work so well on phone. You want PR clean without Claude?
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
What I (personally) want usually is:
name = entry.name to be un-necessary so reducing the diff only required to keep the same code and inlining that access. It didn't mean to rewrite all the occurrences with entry.name.Our policy is that AI is a tool that can be used but it's not something meant to generate PRs for users that would later take credit for (I'm not saying you are, I'm saying that just how AI PRs usually go: most of the time, people just want to have a contribution and so they simply leave their agent in automode).
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
So, the problem generally is not the AI, it's more that whenever there is an LLM, that LLM tends to choose things that are either useless, weird, and that would have been avoided if a human had directly written the lines. If you prefer writing your code through Claude, it's ok, but keep in mind that reviewers are not responsible for guiding the AI, it's the contributor. If we keep telling their agent through our review "no, not like that", it's as if we were directly using an agent in the least efficient way, and hence we are both losing our time.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Ah. FWIW, Claude pointed out that your suggestion resulted in a NameError at string concat below, so it proposed this as easiest way to avoid name global and I accepted it
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Oh, my bad actually it should have been displayname = linkname = name = entry.name. But yeah, in this case, I would have preferred to know why it decided to do so. And then I would have suggested:
displayname = linkname = name = entry.name
instead of the two lines. That's why I prefer that you tell me this first before changing otherwise I would get confused.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
So to reiterate: there is nothing wrong with using Claude (the only think I don't like about it is the long summaries in every PR and trivial things it reports like "all tests pass" which is something I would expect for a PR otherwise it doesn't make sense). It's just that having a 3rd actor usually creates more back and forth between the reviewer and the reviewee.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Co-authored-by: Bénédikt Tran <[email protected]>
The applied suggestion introduced CRLF on the edited lines; normalize them back to LF so the mixed-line-ending check passes.
|
I have made the requested changes; please review again |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Thanks for making the requested changes! @picnixz: please review the changes made to this pull request. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
This PR is stale because it has been open for 90 days with no activity. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.