Description

barneygale

opened on Feb 6, 2024

Issue body actions

In #104512 we made pathlib.Path.glob() use a "walk-and-filter" strategy for expanding ** wildcards in patterns: when we encounter a ** segment, we immediately consume subsequent segments and use them to build a regex that is used to filter results. This saves a bunch of scandir() calls.

However! We actually build a regex for the entire pattern given to glob(), rather than just the segments following ** wildcards. And so when evaluating a pattern like dir*/**/file*, the dir* part is needlessly matched twice against each path. @zooba noted this in a review comment at the time.

We should be able to improve performance by building an re.Pattern only for segments following ** wildcards, and not the entire glob() pattern.

Linked PRs

• GH-115060: Speed up pathlib.Path.glob() by removing redundant regex matching #115061
• GH-115060: Speed up pathlib.Path.glob() by skipping directory scanning #116152
• GH-115060: Speed up pathlib.Path.glob() by not scanning literal parts #117732
• GH-115060: Speed up pathlib.Path.glob() by omitting initial stat() #117831

Activity

1. barneygale

   added

   performancePerformance or resource usagePerformance or resource usage

   topic-pathlib

   on Feb 6, 2024
2. barneygale

   added 2 commits that reference this issue on Feb 6, 2024

   pythonGH-115060: Speed up `pathlib.Path.glob()` by removing redundant…

   54c5aa5

   GH-115060: Speed up `pathlib.Path.glob()` by removing redundant regex…

   6f93b4d
3. barneygale

   closed this as completedon Feb 10, 2024
4. fsc-eriker

   added a commit that references this issue on Feb 14, 2024

   pythonGH-115060: Speed up `pathlib.Path.glob()` by removing redundant…

   ee8ded0
5. barneygale commented on Feb 26, 2024

   

   barneygale

   on Feb 26, 2024

   ContributorAuthor

   More actions

   Re-opening: there's another optimization possible.

   Previous versions of pathlib used path.exists() or path.is_dir() to check whether non-wildcard segments exist, rather than calling path._scandir() on the parent and filtering through a regex. This was dropped when we added the case_sensitive argument.

   If we can somehow check that the underlying filesystem is case-sensitive, and the user sets case_sensitive=True (or leaves it at the default of case_sensitive=None on POSIX), then we can restore this more direct check for existence. And we can do better! For patterns like */foo/bar.py, we can skip checking whether foo is a directory, because we're going to check foo/bar.py exists anyway. Likewise for foo/*, where the _scandir() call failing makes an is_dir() check irrelevant.

   Alternatively, we could add a normalize_case argument to glob(), defaulting to True. If set to False, and case_sensitive is effectively False, then literal segments in results wouldn't be case-normalized.

   Alternatively alternatively, we could enable this behaviour specifically when case_sensitive is set to None (the default), and so users would need to set case_sensitive=False to get fully case-normalized results.

   I need to think about this more.
6. barneygale

   reopened this on Feb 26, 2024
7. barneygale

   added a commit that references this issue on Feb 29, 2024

   pythonGH-115060: Speed up `pathlib.Path.glob()` by skipping directory…

   f1b5603
8. barneygale

   closed this as completedon Apr 6, 2024
9. barneygale commented on Apr 6, 2024

   

   barneygale

   on Apr 6, 2024

   ContributorAuthor

   More actions

   Closed because I'm planning to make pathlib use the glob module for globbing, see #116392
10. barneygale commented on Apr 6, 2024

    

    barneygale

    on Apr 6, 2024

    ContributorAuthor

    More actions

    Re-opening because I'm a prize eejit.
11. barneygale

    reopened this on Apr 6, 2024
12. barneygale

    added 4 commits that reference this issue on Apr 6, 2024

    pythonGH-117586: Speed up `pathlib.Path.glob()` by working with strings

    fe2c46c

    GH-117586: Speed up `pathlib.Path.glob()` by working with strings (#1…

    6258844

    pythonGH-115060: Speed up `pathlib.Path.glob()` by not scanning liter…

    f1b3b0e

    GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal par…

    0eb52f5
13. barneygale

    closed this as completedon Apr 12, 2024
14. barneygale

    reopened this on Apr 13, 2024
15. barneygale commented on Apr 13, 2024

    

    barneygale

    on Apr 13, 2024

    ContributorAuthor

    More actions

    Sorry for the close/open spam, I keep spotting things to do.

    Reacted by Erlend E. Aasland
16. barneygale

    added 2 commits that reference this issue on Apr 13, 2024

    pythonGH-115060: Speed up `pathlib.Path.glob()` by omitting initial `…

    3d1bbe1

    GH-115060: Speed up `pathlib.Path.glob()` by omitting initial `stat()` (

    a74f117
17. barneygale

    closed this as completedon Apr 13, 2024
18. diegorusso

    added 3 commits that reference this issue on Apr 17, 2024

    pythonGH-117586: Speed up `pathlib.Path.glob()` by working with strin…

    99d331b

    pythonGH-115060: Speed up `pathlib.Path.glob()` by not scanning liter…

    a3051d9

    pythonGH-115060: Speed up `pathlib.Path.glob()` by omitting initial `…

    4477521

Issue actions