GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-81079: Add case_sensitive argument to `pathlib.Path.glob()` by barneygale · Pull Request #102710 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Mar 15, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This argument allows case-sensitive matching to be enabled on Windows, and case-insensitive matching to be enabled on Posix.

This PR removes _PreciseSelector and uses _WildcardSelector to select non-wildcard patterns. This allows case sensitivity rules to be varied, and ensures the paths returned from glob() use filesystem casing.

  • Issue: Add parameter @case_sensitive to glob and rglob in pathlib #81079

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@barneygale
pythonGH-81079: Add case_sensitive argument to pathlib.Path.glob()
b0d836e
This argument allows case-sensitive matching to be enabled on Windows, and
case-insensitive matching to be enabled on Posix.
@bedevere-bot bedevere-bot mentioned this pull request Mar 15, 2023
Add parameter @case_sensitive to glob and rglob in pathlib #81079
Closed
@bedevere-bot bedevere-bot added the awaiting review label Mar 15, 2023
@barneygale barneygale changed the title GH-81079: Add case_sensitive argument to pathlib.Path.glob() GH-81079: Add case_sensitive argument to pathlib.Path.glob() Mar 15, 2023
@barneygale
barneygale marked this pull request as ready for review March 15, 2023 01:08
@barneygale barneygale mentioned this pull request Mar 15, 2023
GH-77609: Add follow_symlinks argument to pathlib.Path.glob() #102616
Merged
@barneygale barneygale added the topic-pathlib label Mar 16, 2023
@barneygale barneygale mentioned this pull request Mar 21, 2023
Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
Closed
barneygale added 3 commits March 22, 2023 14:46
@barneygale
Merge branch 'main' into pythongh-81079-glob-case-sensitive-arg-2
e386a54
@barneygale
Merge branch 'main' into pythongh-81079-glob-case-sensitive-arg-2
29cdbee
@barneygale
Merge branch 'main' into pythongh-81079-glob-case-sensitive-arg-2
ac1fe16
@barneygale

barneygale commented Apr 29, 2023

Copy link
Copy Markdown
Contributor Author

Hey @zooba, would you be available to review this? I think you already looked at a similar patch in #102616, so you may already have some useful context. No worries if not. Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba

zooba commented May 2, 2023

Copy link
Copy Markdown
Member

It might be nice to describe the =None value as meaning "using the native case comparison rules for the filesystem". Windows allows marking directories as case-sensitive, and one day we might actually respect that. Similarly, pretty sure you can use FAT from any OS, and we ought to treat that as case insensitive by default.

We don't have to implement those yet, but let's not lock ourselves out by documenting the specific rules. We can also file a feature request now to implement proper case support - it might help find someone motivated to make it happen.

Also, like with the symlink change, I'd like to see clearly stated which parts of the pattern are affected, as I assume it only applies to segments from the first wildcard (that is, foo/bar*/baz with case_sensitive=False on a case sensitive drive matches foo/BARF/BAZ but not FOO/barf/baz).

I trust you (and the tests) on the implementation. It takes me a good few hours to get this code into my head in a way I can judge it, and I don't have time for that today, but also don't want to block on it when all the tests look good :)

barneygale reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale

barneygale commented May 2, 2023

Copy link
Copy Markdown
Contributor Author

Thanks for taking a look!

It might be nice to describe the =None value as meaning "using the native case comparison rules for the filesystem". Windows allows marking directories as case-sensitive, and one day we might actually respect that. Similarly, pretty sure you can use FAT from any OS, and we ought to treat that as case insensitive by default.

We don't have to implement those yet, but let's not lock ourselves out by documenting the specific rules. We can also file a feature request now to implement proper case support - it might help find someone motivated to make it happen.

Good shout, will do!

Also, like with the symlink change, I'd like to see clearly stated which parts of the pattern are affected, as I assume it only applies to segments from the first wildcard (that is, foo/bar*/baz with case_sensitive=False on a case sensitive drive matches foo/BARF/BAZ but not FOO/barf/baz).

It affects all parts of the pattern! So it would match both your examples. I think this is the behaviour users expect when passing case_sensitive=False.

(The same thing will apply when we add support for follow_symlinks=True and follow_symlinks=False -- it will affect all parts of the pattern uniformly)

I trust you (and the tests) on the implementation. It takes me a good few hours to get this code into my head in a way I can judge it, and I don't have time for that today, but also don't want to block on it when all the tests look good :)

I'll expand the PR description to (hopefully) make it easier for you (or anyone else) to get into the patch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
Describe behaviour when case_sensitive is None.
dc82494
@zooba

zooba commented May 2, 2023

Copy link
Copy Markdown
Member

It affects all parts of the pattern! So it would match both your examples. I think this is the behaviour users expect when passing case_sensitive=False.

(The same thing will apply when we add support for follow_symlinks=True and follow_symlinks=False -- it will affect all parts of the pattern uniformly)

I was thinking of the =None case, especially since for case_sensitive that's a legitimate and arguably more useful value (whereas for symlinks it's deprecated/legacy). But applying to the whole pattern, even if the first few parts are constants, seems fine.

barneygale reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale

barneygale commented May 2, 2023

Copy link
Copy Markdown
Contributor Author

While thinking about how to explain this PR, I realised it could be broken down further. So here's two new smallish PRs:

  • GH-104102: Optimize pathlib.Path.glob() handling of ../ pattern segments #104103
  • GH-104104: Optimize pathlib.Path.glob() by avoiding repeated calls to os.path.normcase() #104105

I'll mark this one as a draft for now.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
barneygale marked this pull request as draft May 2, 2023 20:11
barneygale added 4 commits May 3, 2023 00:47
@barneygale
Merge branch 'main' into pythongh-81079-glob-case-sensitive-arg-2
c411cfa
@barneygale
Restore _PreciseSelector
d097f8c
@barneygale
Revert "Restore _PreciseSelector"
33b435d
This reverts commit d097f8c.
@barneygale
Remove unused _is_wildcard_pattern() function
203e734
@barneygale
barneygale marked this pull request as ready for review May 3, 2023 01:19
@barneygale barneygale mentioned this pull request May 3, 2023
Optimize pathlib.Path.glob() by avoiding repeated calls to os.path.normcase() #104104
Closed
@barneygale

barneygale commented May 3, 2023

Copy link
Copy Markdown
Contributor Author

One more for good luck:

  • GH-104114: Fix pathlib.WindowsPath.glob() use of literal pattern segment case #104116
AlexWaygood reacted with laugh emoji
All reactions
  • 😄 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
barneygale marked this pull request as draft May 3, 2023 02:22
@barneygale
Merge branch 'main' into pythongh-81079-glob-case-sensitive-arg-2
37d2e20
@barneygale
barneygale marked this pull request as ready for review May 3, 2023 16:33
@barneygale

barneygale commented May 3, 2023

Copy link
Copy Markdown
Contributor Author

If we'd like to add a case_sensitive argument to PurePath.match() too, then this PR ought to land first:

  • GH-73435: Implement recursive wildcards in pathlib.PurePath.match() #101398
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

zooba
zooba reviewed May 3, 2023
View reviewed changes
Comment thread Doc/library/pathlib.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/pathlib.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

barneygale and others added 2 commits May 3, 2023 18:56
@barneygale @zooba
Update Doc/library/pathlib.rst
2146b2b
Co-authored-by: Steve Dower <[email protected]>
@barneygale
Pass flavour to selector initialiser
b2c4b47
zooba
zooba approved these changes May 4, 2023
View reviewed changes

@zooba zooba left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

barneygale reacted with hooray emoji
All reactions
  • 🎉 1 reaction
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels May 4, 2023
@barneygale
Improve consistency between glob() and rglob() docs
0d697ad
@barneygale
barneygale enabled auto-merge (squash) May 4, 2023 16:37
@barneygale
barneygale merged commit 8100be5 into python:main May 4, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 4, 2023
carljm added a commit to carljm/cpython that referenced this pull request May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2
8a78a36
* main: (61 commits)
  pythongh-64595: Argument Clinic: Touch source file if any output file changed (python#104152)
  pythongh-64631: Test exception messages in cloned Argument Clinic funcs (python#104167)
  pythongh-68395: Avoid naming conflicts by mangling variable names in Argument Clinic (python#104065)
  pythongh-64658: Expand Argument Clinic return converter docs (python#104175)
  pythonGH-103092: port `_asyncio` freelist to module state (python#104196)
  pythongh-104051: fix crash in test_xxtestfuzz with -We (python#104052)
  pythongh-104190: fix ubsan crash (python#104191)
  pythongh-104106: Add gcc fallback of mkfifoat/mknodat for macOS (pythongh-104129)
  pythonGH-104142: Fix _Py_RefcntAdd to respect immortality (pythonGH-104143)
  pythongh-104112: link from cached_property docs to method-caching FAQ (python#104113)
  pythongh-68968: Correcting message display issue with assertEqual (python#103937)
  pythonGH-103899: Provide a hint when accidentally calling a module (pythonGH-103900)
  pythongh-103963: fix 'make regen-opcode' in out-of-tree builds (python#104177)
  pythongh-102500: Add PEP 688 and 698 to the 3.12 release highlights (python#104174)
  pythonGH-81079: Add case_sensitive argument to `pathlib.Path.glob()` (pythonGH-102710)
  pythongh-91896: Deprecate collections.abc.ByteString (python#102096)
  pythongh-99593: Add tests for Unicode C API (part 2) (python#99868)
  pythongh-102500: Document PEP 688 (python#102571)
  pythongh-102500: Implement PEP 688 (python#102521)
  pythongh-96534: socketmodule: support FreeBSD divert(4) socket (python#96536)
  ...
@zhan9san zhan9san mentioned this pull request Feb 18, 2024
Allow cases in ArtifactoryPath.glob() pattern devopshq/artifactory#429
Merged
@SirVer SirVer mentioned this pull request Sep 5, 2026
Work around globbing performance in Python 3.12 SirVer/ultisnips#1696
Merged
SirVer added a commit to SirVer/ultisnips that referenced this pull request Sep 6, 2026
@SirVer
Work around globbing performance in Python 3.12 (#1696)
81d7ceb
`find_all_snippet_directories()` expands the wildcards Vim allows in `'runtimepath'` entries. Since the pathlib migration (c84bb5a) it did so with `Path(pth.anchor).glob(str(pth.relative_to(pth.anchor)))`, i.e. it globbed the whole path starting at `/`. 

Python 3.12's pathlib has no fast path for literal pattern components: every component goes through  _WildcardSelector`, which `scandir`s the parent directory and calls `is_dir()` on each entry. With the ~100 entries of a plugin-heavy runtimepath that lists `/`, `/home`, `/home/user`, ... once per entry (595 directory listings and 5763 stats in the report), and on WSL with symlinked network folders the first snippet expansion froze Vim for seconds. 

Neither 3.11 nor 3.13 have this issue. 3.11 pathlib resolved literal pattern segments through a single `stat`. Python 3.12 removed that path in python/cpython#102710. Python 3.13 brought a literal fast path back in the glob rewrite (python/cpython#117732).

We work around this by finding globbing in the snippet paths and only start globbing from the segment that has globbing characters. While strictly only needed for 3.12 it does not hurt on other Pythons. 

Fixes #1694.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zooba zooba zooba approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@barneygale @zooba @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.