GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-117586: Speed up `pathlib.Path.glob()` by working with strings by barneygale · Pull Request #117589 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Apr 6, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Move pathlib globbing implementation into a new private class: glob._Globber. This class implements fast string-based globbing. It's called by pathlib.Path.glob(), which then converts strings back to path objects.

In the private pathlib ABCs, add a pathlib._abc.Globber subclass that works with PathBase objects rather than strings, and calls user-defined path methods like PathBase.stat() rather than os.stat().

This sets the stage for two more improvements:

  • Speed up pathlib.Path.glob() by removing redundant regex matching #115060
  • Speed up glob.glob() by reducing number of system calls made #116380

Timings:

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*'))"
1000 loops, best of 5: 392 usec per loop
1000 loops, best of 5: 365 usec per loop
# --> 1.07x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*.py'))"
1000 loops, best of 5: 393 usec per loop
1000 loops, best of 5: 371 usec per loop
# --> 1.06x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**'))"
50 loops, best of 5: 9.46 msec per loop
50 loops, best of 5: 9.06 msec per loop
# --> 1.04x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/'))"
50 loops, best of 5: 4.98 msec per loop
50 loops, best of 5: 5.15 msec per loop
# --> 1.03x slower (!)

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/*'))"
20 loops, best of 5: 14 msec per loop
20 loops, best of 5: 12.9 msec per loop
# --> 1.09x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/*.py'))"
20 loops, best of 5: 12.2 msec per loop
20 loops, best of 5: 11.4 msec per loop
# --> 1.07x faster

  • Issue: Speed up pathlib.Path.glob() by working with strings internally #117586

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@barneygale
pythonGH-117586: Speed up pathlib.Path.glob() by working with strings
fe2c46c
Move pathlib globbing implementation to a new module and class:
`pathlib._glob.Globber`. This class implements fast string-based globbing.
It's called by `pathlib.Path.glob()`, which then converts strings back to
path objects.

In the private pathlib ABCs, add a `pathlib._abc.Globber` subclass that
works with `PathBase` objects rather than strings, and calls user-defined
path methods like `PathBase.stat()` rather than `os.stat()`.

This sets the stage for two more improvements:

- pythonGH-115060: Query non-wildcard segments with `lstat()`
- pythonGH-116380: Move `pathlib._glob` to `glob` (unify implementations).
@barneygale barneygale added performance Performance or resource usage topic-pathlib labels Apr 6, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Apr 6, 2024
Speed up pathlib.Path.glob() by working with strings internally #117586
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Apr 6, 2024
@barneygale
Move class into glob module.
26d2c03
@barneygale

barneygale commented Apr 6, 2024

Copy link
Copy Markdown
Contributor Author

This is the first PR in a series that will hopefully unify the globbing implementations in the pathlib and glob modules, and speed both up in the process.

AlexWaygood reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale barneygale mentioned this pull request Apr 6, 2024
GH-116380: Speed up glob.glob() by removing some system calls #116392
Merged
barneygale added 4 commits April 6, 2024 22:22
@barneygale
Fix handling of missing root path.
d6314ac
@barneygale
More precise error handling
8696ca0
@barneygale
Ensure results are normalized.
824f1f6
@barneygale
Speed up results normalization
60eb3d0
@barneygale

barneygale commented Apr 7, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Hey @serhiy-storchaka, does this PR look alright to you? Not requesting a detailed review, more of a sanity check, given you've looked after the glob module for the last few years.

This PR doesn't affect glob.[i]glob(), but it does move pathlib's globbing implementation into glob.py.

Thank you.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

barneygale added 2 commits April 7, 2024 16:43
@barneygale
Define add_slash() in _Globber itself.
98dea96
@barneygale
Slightly speed up path renormalisation.
ebcd7fc
@barneygale

barneygale commented Apr 10, 2024

Copy link
Copy Markdown
Contributor Author

I'll merge this now as it's important for #115060, which I'm hoping to get done in time for 3.13 beta 1.

But I'll leave glob.glob() and glob.iglob() unchanged in 3.13; any PRs I make will target 3.14.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
barneygale merged commit 6258844 into python:main Apr 10, 2024
@bedevere-app bedevere-app Bot removed the awaiting core review label Apr 10, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Apr 10, 2024
@barneygale
pythonGH-117586: Speed up pathlib.Path.walk() by working with strings
0a3d4ff
Move `pathlib.Path.walk()` implementation into `glob._Globber`. The new
`glob._Globber.walk()` classmethod works with strings internally, which is
a little faster than generating `Path` objects and keeping them normalized.
The `pathlib.Path.walk()` method converts the strings back to path objects.

In the private pathlib ABCs, our existing subclass of `_Globber` ensures
that `PathBase` instances are used throughout.

Follow-up to python#117589.
@barneygale barneygale mentioned this pull request Apr 10, 2024
GH-117586: Speed up pathlib.Path.walk() by working with strings #117726
Merged
barneygale added a commit that referenced this pull request Apr 11, 2024
@barneygale
GH-117586: Speed up pathlib.Path.walk() by working with strings (#1…
0cc71bd
…17726)

Move `pathlib.Path.walk()` implementation into `glob._Globber`. The new
`glob._Globber.walk()` classmethod works with strings internally, which is
a little faster than generating `Path` objects and keeping them normalized.
The `pathlib.Path.walk()` method converts the strings back to path objects.

In the private pathlib ABCs, our existing subclass of `_Globber` ensures
that `PathBase` instances are used throughout.

Follow-up to #117589.
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@barneygale @diegorusso
pythonGH-117586: Speed up pathlib.Path.glob() by working with strin…
99d331b
…gs (python#117589)

Move pathlib globbing implementation into a new private class: `glob._Globber`. This class implements fast string-based globbing. It's called by `pathlib.Path.glob()`, which then converts strings back to path objects.

In the private pathlib ABCs, add a `pathlib._abc.Globber` subclass that works with `PathBase` objects rather than strings, and calls user-defined path methods like `PathBase.stat()` rather than `os.stat()`.

This sets the stage for two more improvements:

- pythonGH-115060: Query non-wildcard segments with `lstat()`
- pythonGH-116380: Unify `pathlib` and `glob` implementations of globbing.

No change to the implementations of `glob.glob()` and `glob.iglob()`.
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@barneygale @diegorusso
pythonGH-117586: Speed up pathlib.Path.walk() by working with strin…
ab75678
…gs (python#117726)

Move `pathlib.Path.walk()` implementation into `glob._Globber`. The new
`glob._Globber.walk()` classmethod works with strings internally, which is
a little faster than generating `Path` objects and keeping them normalized.
The `pathlib.Path.walk()` method converts the strings back to path objects.

In the private pathlib ABCs, our existing subclass of `_Globber` ensures
that `PathBase` instances are used throughout.

Follow-up to python#117589.
cjwatson added a commit to cjwatson/pypandoc that referenced this pull request Dec 8, 2024
@cjwatson
Fix convert_file for Python 3.13
bf7bc86
As of python/cpython#117589 (at least),
`Path.glob` returns an `Iterator` rather than `Generator` (which
inherits from `Iterator`).  `convert_file` doesn't need to care about
this distinction; it can reasonably accept both.

This previously caused a test failure along these lines:

  ______________________________________________________ TestPypandoc.test_basic_conversion_from_file_pattern_pathlib_glob _______________________________________________________

  self = <tests.TestPypandoc testMethod=test_basic_conversion_from_file_pattern_pathlib_glob>

      def test_basic_conversion_from_file_pattern_pathlib_glob(self):
          received_from_str_filename_input = pypandoc.convert_file("./*.md", 'html').lower()
  >       received_from_path_filename_input = pypandoc.convert_file(Path(".").glob("*.md"), 'html').lower()

  tests.py:654:
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  source_file = <map object at 0x7f83952c9420>, to = 'html', format = None, extra_args = (), encoding = 'utf-8', outputfile = None, filters = None, verify_format = True
  sandbox = False, cworkdir = '/home/cjwatson/src/python/pypandoc', sort_files = True
  [...]
          if not _identify_path(discovered_source_files):
  >           raise RuntimeError("source_file is not a valid path")
  E           RuntimeError: source_file is not a valid path

  pypandoc/__init__.py:201: RuntimeError
@cjwatson cjwatson mentioned this pull request Dec 8, 2024
Fix convert_file for Python 3.13 JessicaTegner/pypandoc#384
Merged
cjwatson added a commit to cjwatson/typeshed that referenced this pull request Dec 8, 2024
@cjwatson
Weaken return type of Path.{glob,rglob} in 3.13
9674ec2
Since python/cpython#117589 (at least),
`Path.glob` and `Path.rglob` return an `Iterator` rather than a
`Generator`.
cjwatson added a commit to cjwatson/typeshed that referenced this pull request Dec 8, 2024
@cjwatson
Weaken return type of Path.{glob,rglob} in 3.13
795336d
Since python/cpython#117589 (at least),
`Path.glob` and `Path.rglob` return an `Iterator` rather than a
`Generator`.
@cjwatson cjwatson mentioned this pull request Dec 8, 2024
Weaken return type of Path.{glob,rglob} in 3.13 python/typeshed#13223
Merged
JessicaTegner added a commit to JessicaTegner/pypandoc that referenced this pull request Jan 8, 2025
@cjwatson @JessicaTegner
Fix convert_file for Python 3.13 (#384)
3fd939c
As of python/cpython#117589 (at least),
`Path.glob` returns an `Iterator` rather than `Generator` (which
inherits from `Iterator`).  `convert_file` doesn't need to care about
this distinction; it can reasonably accept both.

This previously caused a test failure along these lines:

  ______________________________________________________ TestPypandoc.test_basic_conversion_from_file_pattern_pathlib_glob _______________________________________________________

  self = <tests.TestPypandoc testMethod=test_basic_conversion_from_file_pattern_pathlib_glob>

      def test_basic_conversion_from_file_pattern_pathlib_glob(self):
          received_from_str_filename_input = pypandoc.convert_file("./*.md", 'html').lower()
  >       received_from_path_filename_input = pypandoc.convert_file(Path(".").glob("*.md"), 'html').lower()

  tests.py:654:
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  source_file = <map object at 0x7f83952c9420>, to = 'html', format = None, extra_args = (), encoding = 'utf-8', outputfile = None, filters = None, verify_format = True
  sandbox = False, cworkdir = '/home/cjwatson/src/python/pypandoc', sort_files = True
  [...]
          if not _identify_path(discovered_source_files):
  >           raise RuntimeError("source_file is not a valid path")
  E           RuntimeError: source_file is not a valid path

  pypandoc/__init__.py:201: RuntimeError

Co-authored-by: Jessica Tegner <[email protected]>
srittau pushed a commit to python/typeshed that referenced this pull request Feb 28, 2025
@cjwatson
Weaken return type of Path.{glob,rglob} in 3.13 (#13223)
8ebf8af
Since python/cpython#117589 (at least),
`Path.glob` and `Path.rglob` return an `Iterator` rather than a
`Generator`.
@tungol tungol mentioned this pull request Mar 7, 2025
add new pathlib base classes for 3.13 python/typeshed#12937
Draft
mmingyu pushed a commit to mmingyu/typeshed that referenced this pull request May 16, 2025
@cjwatson mingyu.park
Weaken return type of Path.{glob,rglob} in 3.13 (python#13223)
19f53f6
Since python/cpython#117589 (at least),
`Path.glob` and `Path.rglob` return an `Iterator` rather than a
`Generator`.
@kaitj kaitj mentioned this pull request Jun 11, 2025
Follow symlinks recursively childmindresearch/bids2table#56
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@barneygale
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.