GeezerVison is showing github.com — GH-102613: Improve performance of `pathlib.Path.rglob()` by barneygale · Pull Request #104244 · python/cpython
Conversation
barneygale
commented
May 6, 2023
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Stop de-duplicating results in _RecursiveWildcardSelector. A new _DoubleRecursiveWildcardSelector class is introduced which performs de-duplication, but this is used only for patterns with multiple non-adjacent ** segments, such as path.glob('**/foo/**'). By avoiding the use of a set in most cases, PurePath.__hash__() is not called, and so paths do not need to be parsed and (case-) normalised.
Also merge adjacent ** segments in patterns.
Timings:
$ ./python -m timeit -s 'from pathlib import Path; p = Path()' 'list(p.glob("**/*"))'
1 loop, best of 5: 197 msec per loop # before
2 loops, best of 5: 146 msec per loop # after
--> 35% faster
$ ./python -m timeit -s 'from pathlib import Path; p = Path()' 'list(p.glob("**/**/*"))'
1 loop, best of 5: 1.77 sec per loop # before
2 loops, best of 5: 146 msec per loop # after
--> 12x faster
$ ./python -m timeit -s 'from pathlib import Path; p = Path()' 'list(p.glob("**/*/**"))'
1 loop, best of 5: 738 msec per loop # before
1 loop, best of 5: 731 msec per loop # after
--> about the same
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Stop de-duplicating results in `_RecursiveWildcardSelector`. A new
`_DoubleRecursiveWildcardSelector` class is introduced which performs
de-duplication, but this is used _only_ for patterns with multiple
non-adjacent `**` segments, such as `path.glob('**/foo/**')`. By avoiding
the use of a set, `PurePath.__hash__()` is not called, and so paths do not
need to be parsed and (case-) normalised.
Also merge adjacent '**' segments in patterns.
Uh oh!
There was an error while loading. Please reload this page.
jbower-fb
pushed a commit
to jbower-fb/cpython
that referenced
this pull request
May 8, 2023
…nGH-104244)
Stop de-duplicating results in `_RecursiveWildcardSelector`. A new
`_DoubleRecursiveWildcardSelector` class is introduced which performs
de-duplication, but this is used _only_ for patterns with multiple
non-adjacent `**` segments, such as `path.glob('**/foo/**')`. By avoiding
the use of a set, `PurePath.__hash__()` is not called, and so paths do not
need to be stringified and case-normalised.
Also merge adjacent '**' segments in patterns.
carljm
added a commit
to carljm/cpython
that referenced
this pull request
May 9, 2023
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|