GeezerVison is showing github.com — Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir() · Issue #151788 · python/cpython
Speed up SimpleHTTPRequestHandler.list_directory() by using os.scandir() #151788DescriptionSimpleHTTPRequestHandler.list_directory() calls os.listdir() and then, for every entry, os.path.isdir() (a stat) and os.path.islink() (an lstat) — two stat-family syscalls per entry. This is wasted work on any filesystem and dominates listing time for large directories; on network filesystems like NFS, where each call is a round-trip, it becomes severe.
os.scandir() returns the entry type from the directory read itself (POSIX d_type / NFS READDIRPLUS), eliminating the per-entry stats in the common case. CPython already did this migration for os.walk(), glob, and pathlib.Path.iterdir() (gh-117727); http.server was missed.
Benchmark
Directory with 1000 files + 1000 dirs (plus a few symlinks):
- stat-family syscalls (
strace): 4088 → 88 (the 88 is constant interpreter startup; the per-entry loop drops from ~2 syscalls to ~0)
- local filesystem wall-clock: ~10× faster
- emulating NFS by injecting per-
stat latency: the listing goes from seconds to ~2 ms
Worst case — a mount that returns DT_UNKNOWN — falls back to one cached lstat per entry, which is still fewer calls than today and never worse.
The change is behavior-preserving: DirEntry.is_dir()/is_symlink() match os.path.isdir/os.path.islink semantics (follow-symlinks behavior and return-False-on-error), verified across real dirs/files, symlink-to-dir, symlink-to-file, and broken symlinks. The existing test_httpservers suite passes unchanged.
I have a patch ready and will open a PR.
This issue was prepared with AI assistance (Claude Code); the analysis and benchmarks were reviewed by me.
Linked PRs
Activitypicnixz commented on Jun 20, 2026 I'd like to say that http.server is NOT something that should be used in production, and thus it's ok if we're slow. I really don't want to change http.server more than what's already needed.
That being said, the new code looks a bit cleaner so I'm not really against but some comments are misleading so I'll review that. But please, don't focus on speeding-up components that are not meant for production. picnixz commented on Jun 20, 2026 Actually, I prefer that we don't make that change. With DirEntry objects, we need to catch OSError when checking whether something is a file or not as its methods may raise exceptions while their os.path.* counterparts don't. mjbommar commented on Jun 20, 2026 ContributorAuthor More actions @picnixz I'm sorry, do you mean you don't want the change at all or just part of it you mentioned in the code review?
I get that it's not a production hosting method, but I think it's probably the most common way on earth that people serve ad hoc HTTP traffic in linux since RH/Deb started baking python in. picnixz commented on Jun 20, 2026
think it's probably the most common way on earth that people serve ad hoc HTTP traffic in linux since RH/Deb started baking python in.
And in this case, it's wrong. It's NOT to be used as a production server. Usually, if you already just serve a contents directory you're not worried about performance like that.
I'm sorry, do you mean you don't want the change at all or just part of it you mentioned in the code review?
Considering the fact that we would need to wrap all calls in try-except to catch possible OS errors and then silence them, but I honestly don't think it's really necessary. In addition, when you say "emulating NFS by injecting per-stat latency: the listing goes from seconds to ~2 ms", then I don't think it's really necessary. How much latency did you add? I don't see why having a cached approach would be better in this case. I also don't understand why there is no statcall at all for this changed part of the code. mjbommar commented on Jun 20, 2026 ContributorAuthor More actions
think it's probably the most common way on earth that people serve ad hoc HTTP traffic in linux since RH/Deb started baking python in.
And in this case, it's wrong. It's NOT to be used as a production server. Usually, if you already just serve a contents directory you're not worried about performance like that.
My point is only that millions of people are probably affected. E.g., I know many labs use this inside of something like a RHEL bootstrap to serve PXE boot images. There is no other option inside of a bare Debian/RHEL without installing third party dependencies.
Considering the fact that we would need to wrap all calls in try-except to catch possible OS errors and then silence them, but I honestly don't think it's really necessary. In addition, when you say "emulating NFS by injecting per-stat latency: the listing goes from seconds to ~2 ms", then I don't think it's really necessary. How much latency did you add? I don't see why having a cached approach would be better in this case. I also don't understand why there is no statcall at all for this changed part of the code.
It's a real performance gain on any FS, especially NFS. I tested on multiple real NAS devices but don't have a stable, quiet device to test, hence this approach.
To be honest, if you are opposed to a QoL improvement here, I think it begs the question whether you should just deprecate the functionality.
Either keep it and improve it or drop it and force vendors/etc. to find another solution. Everyone would be better off compared to status quo. picnixz commented on Jun 20, 2026
To be honest, if you are opposed to a QoL improvement here, I think it begs the question whether you should just deprecate the functionality.
I don't understand this argument. Your QoL improvement only relies on simulation. When you say:
It's a real performance gain on any FS, especially NFS. I tested on multiple real NAS devices but don't have a stable, quiet device to test, hence this approach.
What are the numbers in this case? I don't care whether it's quiet or not, because even if it's a bit noisy system, we should still see differences. You claim an improvement that goes from seconds to milliseconds. In addition, the list_directory is for listing the current directory and your tests considers 1000+ files and 1000 directories. I have NOT seen a production server serving something with thousand files and directories in a single directory (and even if it exists, I don't think it will be the bottlneck for users)
Either keep it and improve it or drop it and force vendors/etc. to find another solution. Everyone would be better off compared to status quo.
I'm not against QoL improvements but I'm against improvements that are not clearly shown. Until now, I have not seen any real numbers except about simulation. So, unless there is a real production change, I don't think it's necessary to change the code. I still want to know why we only have 88 statcalls, as if there were... nothing to stat at all? mjbommar commented on Jun 20, 2026 ContributorAuthor More actions The difference was like >1000x on my loaded NAS devices, which I thought would look unbelievable. The reduced round-tripping on networked filesystems is huge, especially when you saturate a queue.
Here's a microbenchmark you can run on your own system. On my Linux boxes, even on tmpfs, it's a big speedup. You can see Win11 numbers too, so it's durable win across OS.
OS / FS / Python | entries | listdir | scandir | speedup
--------------------------------+---------+-----------+-----------+--------
Linux tmpfs (RAM) py3.14 | 105 | 0.37 ms | 0.04 ms | 8.6x
Linux tmpfs (RAM) py3.14 | 1,050 | 3.73 ms | 0.38 ms | 9.9x
Linux tmpfs (RAM) py3.14 | 21,000 | 83.76 ms | 10.99 ms | 7.6x
Linux ext4 (disk) py3.14 | 105 | 0.38 ms | 0.05 ms | 7.4x
Linux ext4 (disk) py3.14 | 1,050 | 3.81 ms | 0.51 ms | 7.4x
Linux ext4 (disk) py3.14 | 22,050 | 93.36 ms | 14.59 ms | 6.4x
Windows 11 NTFS py3.13 | 105 | 1.98 ms | 0.59 ms | 3.4x
Windows 11 NTFS py3.13 | 1,050 | 15.69 ms | 2.93 ms | 5.4x
Windows 11 NTFS py3.13 | 21,000 | 391.33 ms | 79.91 ms | 4.9x
Microbenchmark
#!/usr/bin/env python3
"""Micro-benchmark: http.server list_directory(), os.listdir vs os.scandir.
Self-contained, stdlib only, no simulation. Both functions below mirror exactly
the type-detection work SimpleHTTPRequestHandler.list_directory() does (the only
part that touches the filesystem). The new design uses os.scandir() and handles
OSError the same way os.path.isdir()/islink() do (return False), so behavior is
identical -- the `verify` subcommand asserts that.
Subcommands (so an external harness can isolate one design for strace/cProfile):
python3 bench_list_directory.py sweep [SIZE ...] # wall-clock comparison
python3 bench_list_directory.py make DIR COUNT # build a tree, print path
python3 bench_list_directory.py run old|new DIR # run one design once
python3 bench_list_directory.py verify DIR # assert old == new output
Point TMPDIR at an NFS mount to measure the network-round-trip effect:
TMPDIR=/mnt/nfs python3 bench_list_directory.py sweep
"""
import os
import sys
import time
import tempfile
import shutil
def build_old(path):
"""Current code: os.listdir + os.path.isdir/islink (2 stat-family calls/entry)."""
names = os.listdir(path)
names.sort(key=lambda a: a.lower())
out = []
for name in names:
fullname = os.path.join(path, name)
displayname = linkname = name
if os.path.isdir(fullname):
displayname = linkname = name + "/"
if os.path.islink(fullname):
displayname = name + "@"
out.append((linkname, displayname))
return out
def build_new(path):
"""Proposed code: os.scandir, type from d_type; OSError handled like os.path.*."""
with os.scandir(path) as it:
entries = sorted(it, key=lambda e: e.name.lower())
out = []
for entry in entries:
name = entry.name
displayname = linkname = name
try:
is_dir = entry.is_dir()
except OSError:
is_dir = False
try:
is_link = entry.is_symlink()
except OSError:
is_link = False
if is_dir:
displayname = linkname = name + "/"
if is_link:
displayname = name + "@"
out.append((linkname, displayname))
return out
DESIGNS = {"old": build_old, "new": build_new}
def make_tree(total, into=None):
"""Realistic-ish mix: ~half files, ~half dirs, 5% symlinks."""
d = into or tempfile.mkdtemp(prefix="bench_ld_")
n_dirs = total // 2
n_files = total - n_dirs
n_links = max(1, total // 20)
for i in range(n_files):
open(os.path.join(d, f"file_{i:06}.dat"), "w").close()
for i in range(n_dirs):
os.mkdir(os.path.join(d, f"dir_{i:06}"))
for i in range(n_links):
os.symlink(f"dir_{i:06}", os.path.join(d, f"link_{i:06}"))
return d
def _fstype(path):
try:
import subprocess
return subprocess.check_output(
["stat", "-f", "-c", "%T", path], text=True).strip()
except Exception:
return "?"
def best_of(fn, path, repeat):
t = float("inf")
for _ in range(repeat):
s = time.perf_counter()
fn(path)
t = min(t, time.perf_counter() - s)
return t
def cmd_sweep(argv):
repeat = 7
sizes = [int(a) for a in argv] if argv else [100, 1000, 5000, 20000]
tmp = tempfile.gettempdir()
print(f"python {sys.version.split()[0]} | tmpdir={tmp} ({_fstype(tmp)}) | "
f"best-of-{repeat}, identical-output asserted")
for total in sizes:
d = make_tree(total)
try:
assert build_old(d) == build_new(d), "OUTPUT DIFFERS!"
entries = len(os.listdir(d))
old = best_of(build_old, d, repeat)
new = best_of(build_new, d, repeat)
print(f" {entries:>7} entries | listdir {old*1e3:9.3f} ms | "
f"scandir {new*1e3:9.3f} ms | {old/new:6.1f}x")
finally:
shutil.rmtree(d)
return 0
def cmd_make(argv):
if len(argv) != 2:
sys.exit("usage: make DIR COUNT")
d, count = argv[0], int(argv[1])
os.makedirs(d, exist_ok=True)
make_tree(count, into=d)
print(d)
return 0
def cmd_run(argv):
if len(argv) != 2 or argv[0] not in DESIGNS:
sys.exit("usage: run old|new DIR")
DESIGNS[argv[0]](argv[1])
return 0
def cmd_verify(argv):
if len(argv) != 1:
sys.exit("usage: verify DIR")
ok = build_old(argv[0]) == build_new(argv[0])
print("identical output: %s" % ("OK" if ok else "MISMATCH"))
return 0 if ok else 1
def main():
cmds = {"sweep": cmd_sweep, "make": cmd_make, "run": cmd_run, "verify": cmd_verify}
argv = sys.argv[1:]
cmd = argv[0] if argv and argv[0] in cmds else "sweep"
rest = argv[1:] if argv and argv[0] in cmds else argv
sys.exit(cmds[cmd](rest))
if __name__ == "__main__":
main()
picnixz commented on Jun 20, 2026 Honestly, I'm really not convinced. We're still talking about milliseconds and I doubt anyone will see the difference. So instead of advertising it to go from seconds to ms, it's more that we go from ms to smaller ms. The bottleneck will be in the network, not in listing the contents.
But considering we have already done the changes in other parts of the stdlib, I'd say why not. However, we need to add try-except blocks with fallback to = False so as to emulate the behavior of plain os.path.* functions. Or leave them as is. What happens if you only change the os.scandir part? I still don't understand and you still haven't explained why there are only 88 statcalls (you present this as an argument for the change but I still didn't get an explanation of why this happened and I would appreciate to understand this part specifically). mjbommar commented on Jun 20, 2026 ContributorAuthor More actions
Honestly, I'm really not convinced. We're still talking about milliseconds and I doubt anyone will see the difference. So instead of advertising it to go from seconds to ms, it's more that we go from ms to smaller ms. The bottleneck will be in the network, not in listing the contents.
But considering we have already done the changes in other parts of the stdlib, I'd say why not. However, we need to add try-except blocks with fallback to = False so as to emulate the behavior of plain os.path.* functions. Or leave them as is. What happens if you only change the os.scandir part? I still don't understand and you still haven't explained why there are only 88 statcalls (you present this as an argument for the change but I still didn't get an explanation of why this happened and I would appreciate to understand this part specifically).
Here's my experience.
I have Linux-based NAS devices that expose NFS to my home lab. One device is an 4x20TB RAID0 NAS that serves hundreds of qemu images and source code work trees, e.g., cpython and linux kernel. Sometimes, I want to interactively browse a folder or transfer a file to a host that doens't have scp - hence quick use of uv run python -m http.server ...
When the 7200RPM platters in my NAS hit their IOPS limit, or when the wifi link to my laptop saturates because my kids are on the PS5, every extra stat adds 20-50ms of traffic on the current implementation because the stat calls are queued up on the disk or switch/router.
Now, since I've been working on NFS/SMB patches on the kernel lately, I remembered that there are some big differences between the access patterns. And when you look at Modules/posixmodule.c, you can see that scandir keeps the full OS/FS return data that includes path and type for child entries, but listdir never stores it, requiring the per entry calls. That's how the syscall number drops, because it's effectively batched up front.
Now, there is a subtle TOCTOU change. But then there's also the fact that the current behavior is not a consistent view of the directory, as entries can be added/removed/changed as the existing method loops the stat calls.
I asked Claude to research and summarize across the major OS/FS combos below.
| OS |
Filesystem / mount |
Dir-read method (scandir uses) |
Type in readdir? |
stat()/entry old → new |
Summary |
| Linux |
ext4 / xfs / btrfs / tmpfs |
getdents64 (d_type) |
yes (DT_DIR/REG) |
2 → 0 |
Big win; per-entry stat eliminated (symlinks cost 1, to follow the link) |
| Linux |
NFS w/ READDIRPLUS (e.g. ReadyNAS, default modern Linux client) |
NFS READDIRPLUS → d_type |
yes (from RPC attrs) |
2 → 0 |
Biggest win — each removed stat was a network GETATTR round-trip |
| Linux |
NFS, plain READDIR (nordirplus, or large-dir readdirplus fallback) |
NFS READDIR |
no (DT_UNKNOWN) |
2 → 2 |
Parity, never worse (fallback stat = GETATTR per entry, same as today) |
| Linux |
some FUSE filesystems |
getdents64 |
often no (DT_UNKNOWN) |
2 → 2..0 |
Parity to win, depending on the FS driver |
| macOS / *BSD |
APFS / HFS+ / UFS |
readdir (d_type) |
yes |
2 → 0 |
Win (same mechanism as Linux local) |
| Windows 11 |
NTFS / ReFS |
FindFirstFileW / FindNextFileW |
yes (full attrs) |
2 → 0 |
Win — attributes come with the scan (measured 3.4–5.4× on NTFS) |
picnixz commented on Jun 20, 2026 Ok, so since you see a local improvement, and while I still don't think it's a good idea to use http.server for non-local development in general and consider it a niche case that complicate a bit the code, I believe there may be others in your case (I however doubt that official websites would you this; they generally use apache or nginx for exposing serving a simple HTTP server).
If we put a NEWS entry, we should not claim 10x improvements but rather just say that we switched to using scandir instead of listdir to improve the performance of list_directories on systems with slow statcalls or something like that. If something breaks in the wild because of some other obscure things, it would be good to know what was the cause. But let's not claim 10x improvements when this is not the case.
I should also note that your benchmark only selects the fastest speed and may also rely on the fact that the OS is caching some information. So could you, just for validation purposes, rerun the benchmarks by swapping the order of the tests (first test the scandir() approach and then the listdir() approach). We've seen cases where OS caches were incorrectly showing something being fast (or slow). mjbommar commented on Jun 20, 2026 ContributorAuthor More actions OK, I pushed new commits for this:
About the benchmark, I tested and confirmed it's not about ordering or caching.
ext4 (disk), repeat=9
old-first 1000e | listdir best 3.838 / median 3.862 ms | scandir best 0.529 / median 0.538 ms
scandir-FIRST 1000e | listdir best 3.818 / median 3.831 ms | scandir best 0.526 / median 0.545 ms
old-first 5000e | listdir best 19.70 / median 20.66 ms | scandir best 3.002 / median 3.041 ms
scandir-FIRST 5000e | listdir best 19.42 / median 19.96 ms | scandir best 2.993 / median 3.024 ms
I kept the NEWS entry and toned it down based on your comments here but let me know if you want to just drop it. picnixz commented on Jun 20, 2026 No the NEWWS entry is good like that.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|