GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-73435: Implement recursive wildcards in `pathlib.PurePath.match()` by barneygale · Pull Request #101398 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Jan 28, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

PurePath.match() now handles the ** wildcard as in Path.glob(), i.e. it matches any number of path segments.

We now compile a re.Pattern object for the entire pattern. This is made more difficult by fnmatch not treating directory separators as special when evaluating wildcards (*, ?, etc), and so we arrange the path parts onto separate lines in a string, and ensure we don't set re.DOTALL.

This improves performance of match() around 2x-3x times for simple patterns, and more for complex patterns:

$ ./python -m timeit \
    -s 'from pathlib import PureWindowsPath as P; path = P("C:/foo/bar.py"); pattern = P("c:/*/*.py")' \
    'path.match(pattern)'
50000 loops, best of 5: 8.13 usec per loop   # before
1000000 loops, best of 5: 297 nsec per loop  # after
  • Issue: Support recursive wildcards in pathlib.PurePath.match() #73435

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
woess reacted with thumbs up emoji AlexWaygood reacted with rocket emoji
All reactions
  • 👍 1 reaction
  • 🚀 1 reaction
@barneygale
pythongh-73435: Implement recursive wildcards in pathlib.PurePath.mat…
608e917 
…ch()

Add a new *recursive* argument to `pathlib.PurePath.match()`, defaulting
to `False`. If set to true, `match()` handles the `**` wildcard as in
`Path.glob()`, i.e. it matches any number of path segments.

We now compile a `re.Pattern` object for the entire pattern. This is made
more difficult by `fnmatch` not treating directory separators as special
when evaluating wildcards (`*`, `?`, etc), and so we arrange the path parts
onto separate *lines* in a string, and ensure we don't set `re.DOTALL`.
@bedevere-bot bedevere-bot added the awaiting review label Jan 28, 2023
@bedevere-bot bedevere-bot mentioned this pull request Jan 28, 2023
Support recursive wildcards in pathlib.PurePath.match() #73435
Closed
@barneygale

barneygale commented Jan 28, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Two big caveats:

  • This doesn't support paths or patterns with embedded newlines
  • pathlib.PureWindowsPath.match() mishandles path anchors #101360 must be fixed first
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
Simplify code slightly
9a43c7f
@barneygale barneygale added the topic-pathlib label Feb 1, 2023
barneygale added 8 commits February 15, 2023 18:00
@barneygale
Fix support for newlines
a846279
@barneygale
Cache translation of individual components
bbd8cd6
@barneygale
Drop 'recursive' argument, make this the only behaviour.
b5c002e
@barneygale
Undo modifications to fnmatch.py
0afcd54
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
fe32717
@barneygale
Fix Windows support
7b6f850
@barneygale
Tidy up code.
037488a
@barneygale
Add news blurb.
0741950
@barneygale
barneygale marked this pull request as ready for review February 17, 2023 19:24
@barneygale barneygale added the performance Performance or resource usage label Feb 17, 2023
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
e1c9731
This was referenced Mar 12, 2023
Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
Closed
GH-77609: Add follow_symlinks argument to pathlib.Path.glob() #102616
Merged
'*' matches entire path in fnmatch #72904
Closed
barneygale added 9 commits April 3, 2023 21:02
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
db6f0ad
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
8dff9e2
@barneygale
Simplify patch; prepare for use in glob()
314679f
@barneygale
Make better use of path object caching.
90eebcc
@barneygale
Add performance tip to docs
4b5fffd
@barneygale
Skip re-initialisation of PurePath patterns.
5e8bc28
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
e81ab5a
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
afb8047
@barneygale
Use re.IGNORECASE rather than os.path.normcase()
722a1ab
@barneygale barneygale changed the title gh-73435: Implement recursive wildcards in pathlib.PurePath.match() GH-73435: Implement recursive wildcards in pathlib.PurePath.match() May 3, 2023
barneygale added 9 commits May 14, 2023 21:18
@barneygale
Explain _FNMATCH_SLICE
ced8998
@barneygale
Accidentally a word.
a33c7b6
@barneygale
Cache pattern compilation
4b3bddb
@barneygale
Remove unneeded from None suffix, whoops.
6ad30dd
@barneygale
Tiny performance improvement: avoid accessing path.parts
052890f
@barneygale
Typo fix
d789b6d
@barneygale
Avoid hashing path object when compiling pattern.
4fe77c6
@barneygale
More performance tweaks
4770c13
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
559787d
@barneygale
barneygale requested a review from AlexWaygood May 18, 2023 18:34
barneygale added 5 commits May 23, 2023 23:32
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
9c09fc4
@barneygale
Re-target to 3.13.
eb35dbc
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
8959dfd
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
fec7702
@barneygale
Merge branch 'main' into pythongh-73435-pathlib-match-recursive
89bc380
zooba
zooba reviewed May 30, 2023
View reviewed changes

@zooba zooba left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but consider adding a couple of comments (as suggested) so that the next person who has to trace through this code is grateful rather than mad at you ;-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/pathlib.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/pathlib.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
Add more comments!
9211297
@zooba

zooba commented May 30, 2023

Copy link
Copy Markdown
Member

Perfect! Ship it
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood
AlexWaygood reviewed May 30, 2023
View reviewed changes
Comment thread Lib/pathlib.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@barneygale @AlexWaygood
Update Lib/pathlib.py
73bb309 
Co-authored-by: Alex Waygood <[email protected]>
@barneygale
barneygale enabled auto-merge (squash) May 30, 2023 19:50
@barneygale

barneygale commented May 30, 2023

Copy link
Copy Markdown
Contributor Author

Thank you for your help Alex, Hugo and Steve!
AlexWaygood reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale mentioned this pull request May 30, 2023
Improve performance of pathlib.PurePath.match() #105113
Closed
@barneygale
barneygale merged commit 49f90ba into python:main May 30, 2023
@bedevere-bot bedevere-bot removed the awaiting review label May 30, 2023
@barneygale barneygale mentioned this pull request May 30, 2023
[3.12] GH-105113: Improve performance of pathlib.PurePath.match() #105114
Merged
This was referenced Jul 2, 2023
pathlib.PurePath.match() can incorrectly match empty path #106330
Closed
GH-106330: Fix matching of empty path in pathlib.PurePath.match() #106331
Merged
@barneygale

barneygale commented Jul 16, 2023

Copy link
Copy Markdown
Contributor Author

Hey, if it interests anyone, I have a follow-up PR that simplifies a bunch of the code added in this PR. It does this by adding a new seps parameter to fnmatch.translate(), and varying the generated regular expresison when its supplied.

  • GH-72904: Add glob.translate() function #106703
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale mentioned this pull request Jan 20, 2024
GH-73435: Add pathlib.PurePath.full_match() #114350
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hugovk hugovk hugovk left review comments
@zooba zooba zooba left review comments
@AlexWaygood AlexWaygood AlexWaygood left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@barneygale @hugovk @zooba @AlexWaygood @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.