GeezerVison is showing github.com — Glob behavior is inconsistent across libraries · Issue #106747 · python/cpython
Glob behavior is inconsistent across libraries #106747Description
the [Python] docs for pathlib.glob are unclear on the meaning of **
You are right, and it gets even more confusing:
glob.glob('**') returns dirs and files, non-recursively, not including .glob.glob('**', recursive=True) returns dirs and files, recursively, not including .list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .
I think there should be a Python upstream issue to document all of that, including the differences between glob and pathlib.
Then zipp could implement what's documented for pathlib.
Repro
Python 3.10.11
mkdir -p dir/subdir ; touch toplevel.txt dir/indir.txt creates:
toplevel.txt
dir/
indir.txt
subdir/
>>> import glob, pathlib
>>> glob.glob('**')
['dir', 'toplevel.txt']
>>> glob.glob('**', recursive=True)
['dir', 'dir/subdir', 'dir/indir.txt', 'toplevel.txt']
>>> list(pathlib.Path('.').glob('**'))
[PosixPath('.'), PosixPath('dir'), PosixPath('dir/subdir')]Originally posted by @nh2 in jaraco/zipp#102 (comment)
Linked PRs
Activityjaraco commented on Jul 14, 2023 Without having thought about it too much, I'd argue pathlib.Path.glob('**') should return dirs and files recursively, not including ., same as glob.glob(recursive=True). I'd also argue that glob.glob('**') should imply recursive=True. After all, what's the meaning of ** if not recursive? jaraco commented on Jul 14, 2023 @barneygale Do you have any insight on the concern? Any opinions? jaraco commented on Jul 14, 2023 Another point of reference, ls on Unix also treats ** as recursive without the dot:
# ls **
dir/indir.txt toplevel.txt
dir:
indir.txt subdir
dir/subdir:
barneygale commented on Jul 14, 2023 Yeah, it's a mess
the [Python] docs for pathlib.glob are unclear on the meaning of **
Logged as #101112, stale-ish PR up at #101223
glob.glob('**', recursive=True) returns dirs and files, recursively, not including .
Try glob.glob('tests/**', recursive=True) - you'll find that tests is included in the results, and so ** is capable of consuming zero path segments. The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.
list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .
Logged as #70303, PR up as #105413. jaraco commented on Jul 15, 2023 I'm going to be nitpicky because this issue demands it. Understand my matter of fact tone below may come off as hostile, but it's not intended that way. I'm merely trying to lay out the facts and disentangle the concerns.
you'll find that tests is included in the results
More precisely, tests/ is included in the results:
cpython main @ glob.glob('Lib/test/**', recursive=True)[0]
'Lib/test/'
Note I'm using xonsh shell, which is why you see python interleaved with a shell prompt in the cpython repo.
Interestingly, tests/ is not included in the results if recursive=False.
The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.
That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".
Even pathlib only emits the . when a recursive glob is indicated:
cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')
I'm now confused. These two statements seem contradictory:
tests is included in the results
it doesn't emit .
In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?
Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies? barneygale commented on Jul 15, 2023
That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".
Even pathlib only emits the . when a recursive glob is indicated:
cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')
I was talking about **, not *. The behaviour of standalone * pattern segments is well-defined: it always consumes one path segment.
In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?
I'm saying that these two function calls should return the same result, as they do in pathlib:
>>> def myglob(pattern, root_dir=''):
... for path in glob.glob(pattern, root_dir=root_dir, recursive=True):
... yield os.path.join(root_dir, path)
...
>>> list(myglob('testdir/**', ''))
['testdir/', 'testdir/a', 'testdir/b']
>>> list(myglob('**', 'testdir'))
['testdir/a', 'testdir/b']
Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies?
** is described as matching "any files and zero or more directories". You can see that, in the testdir/** example above, it can consume zero files and zero directories. But in the ** example, it doesn't consume the current directory.
barneygale commented on Jul 15, 2023 barneygale commented on Jul 26, 2023 FYI, I'm looking to add a new sep argument to fnmatch.translate() which would unify its pattern-matching with pathlib.PurePath.match() and pathlib.Path.glob(). In doing so it would make fnmatch.translate() more compatible with glob.glob(), but there would still be differences, such as the above-noted difference in how ** works, but also the default treatment of hidden files/directories. PR here:
jaraco commented on Aug 4, 2023 I'm very interested in reviewing the proposed change. I'm particularly interested to see if it obviates or conflicts with zipp.glob. serhiy-storchaka commented on Mar 5, 2024 Let resolve some misunderstandings.
glob.glob('**') returns dirs and files, non-recursively, not including .
** without recursive=True is interpreted as * (two subsequent *s, but this is the same). This is not an example for **.
Another point of reference, ls on Unix also treats ** as recursive without the dot:
ls does not interpolate patterns. The shell does this. By default ** is equivalent to *, which is expanded to dir toplevel.txt. ls dir produces indir.txt subdir, and ls toplevel.txt produces toplevel.txt. I do not know how did you get your result, except if you use very non-standard shell and ls.
In these two cases but no others, ** effectively means **/*, i.e. the returned paths must have at least one segment. I'm increasingly convinced that this is wrong.
The patch cannot have less than one component.
Also different: glob.glob("", root_dir=path) returns nothing, but path.glob("") raises ValueError. I think they're both wrong, and instead the path should be yielded if it's a directory.
No file or directory name matches pattern "", therefore the glob.glob() result is correct. As for path.glob("") it is a matter of question how much we want it to differ from glob.glob(). They cannot be completely equivalent for objective reasons, so we should decide what differences are deliberate. Raising an error is safer than producing a different result. Reacted by Barney Gale and Kerim Kabirov barneygale commented on Mar 5, 2024 Thanks for the clarifications
The patch cannot have less than one component.
pathlib.Path('.') is a zero-component path in pathlib. Its .parts is an empty tuple, whereas Path('foo').parts is a 1-element tuple.
os.path.relpath() needs to count components in order to generate the correct number of ../ components, and counts empty/dot paths as having zero components:
>>> os.path.relpath('foo', start='.')
'foo'
>>> os.path.relpath('foo', start='bar')
'../foo'barneygale commented on Mar 5, 2024 FWIW, most of these differences highlighted in this thread are now either resolved or documented.
I'll open a PR to document that glob.glob(root_dir=root_dir) never returns an empty path (that could represent the root_dir), whereas pathlib.Path(root_dir).glob() can return the root_dir. I think that'll cover the remaining differences. Reacted by Jason R. Coombs and Maciej Olko
MetadataMetadataStatusShow more project fields StatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|