GeezerVison is showing github.com — GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts by barneygale · Pull Request #117732 · python/cpython
Conversation
barneygale
commented
Apr 10, 2024
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Don't bother calling os.scandir() to scan for literal pattern segments, like foo in foo/*.py. Instead, append the segment(s) as-is and call through to the next selector with exists=False, which signals that the path might not exist. Subsequent selectors will call os.scandir() or os.lstat() to filter out missing paths as needed.
Timings:
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib'))"
5000 loops, best of 5: 69.4 usec per loop
20000 loops, best of 5: 13.6 usec per loop
# --> 5.1x faster
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/'))"
5000 loops, best of 5: 73.3 usec per loop
20000 loops, best of 5: 14.2 usec per loop
# --> 5.16x faster
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*'))"
1000 loops, best of 5: 362 usec per loop
1000 loops, best of 5: 301 usec per loop
# --> 1.2x faster
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*/__init__.py'))"
200 loops, best of 5: 1.18 msec per loop
1000 loops, best of 5: 273 usec per loop
# --> 4.32x faster
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/__init__.py'))"
50 loops, best of 5: 9.46 msec per loop
50 loops, best of 5: 5.72 msec per loop
# --> 1.65x faster
$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/pathlib/__init__.py'))"
1000 loops, best of 5: 210 usec per loop
20000 loops, best of 5: 14.9 usec per loop
# --> 14.1x faster
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
…al parts
Don't bother calling `os.scandir()` to scan for literal pattern segments,
like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call
through to the next selector with `exists=False`, which signals that the
path might not exist. Subsequent selectors will call `os.scandir()` or
`os.lstat()` to filter out missing paths as needed.
diegorusso
pushed a commit
to diegorusso/cpython
that referenced
this pull request
Apr 17, 2024
…al parts (python#117732)
Don't bother calling `os.scandir()` to scan for literal pattern segments,
like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call
through to the next selector with `exists=False`, which signals that the
path might not exist. Subsequent selectors will call `os.scandir()` or
`os.lstat()` to filter out missing paths as needed.
SirVer
added a commit
to SirVer/ultisnips
that referenced
this pull request
Sep 6, 2026
`find_all_snippet_directories()` expands the wildcards Vim allows in `'runtimepath'` entries. Since the pathlib migration (c84bb5a) it did so with `Path(pth.anchor).glob(str(pth.relative_to(pth.anchor)))`, i.e. it globbed the whole path starting at `/`.
Python 3.12's pathlib has no fast path for literal pattern components: every component goes through _WildcardSelector`, which `scandir`s the parent directory and calls `is_dir()` on each entry. With the ~100 entries of a plugin-heavy runtimepath that lists `/`, `/home`, `/home/user`, ... once per entry (595 directory listings and 5763 stats in the report), and on WSL with symlinked network folders the first snippet expansion froze Vim for seconds.
Neither 3.11 nor 3.13 have this issue. 3.11 pathlib resolved literal pattern segments through a single `stat`. Python 3.12 removed that path in python/cpython#102710. Python 3.13 brought a literal fast path back in the glob rewrite (python/cpython#117732).
We work around this by finding globbing in the snippet paths and only start globbing from the segment that has globbing characters. While strictly only needed for 3.12 it does not hurt on other Pythons.
Fixes #1694.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|