Description

domdfcoding

opened on Apr 16, 2022

Issue body actions

Bug report

On Python 3.11.0a7, calling Path.rglob("*.*") returns an iterator over no files, whereas on previous versions (including alpha 6) it would correctly return an iterator over all files in the directory and all of its children.

The following code illustrates this problem:

import pathlib, tempfile, pprint

with tempfile.TemporaryDirectory() as tmpdir:
	tmp_path = pathlib.Path(tmpdir)
	(tmp_path / "a.txt").touch()
	(tmp_path / "b.txt").touch()
	(tmp_path / "c").mkdir()
	(tmp_path / "c" / "d.txt").touch()
	pprint.pprint(list(tmp_path.rglob("*.*")))
	pprint.pprint(list(tmp_path.rglob("*.txt")))

Save this as pathlib_rglob.py.

Output:

$ python3.11 pathlib_rglob.py 
[]
[PosixPath('/tmp/tmpngjo2cyn/a.txt'),
 PosixPath('/tmp/tmpngjo2cyn/b.txt'),
 PosixPath('/tmp/tmpngjo2cyn/c/d.txt')]
$ python3.10 pathlib_rglob.py 
[PosixPath('/tmp/tmpodfbqyhx/a.txt'),
 PosixPath('/tmp/tmpodfbqyhx/b.txt'),
 PosixPath('/tmp/tmpodfbqyhx/c/d.txt')]
[PosixPath('/tmp/tmpodfbqyhx/a.txt'),
 PosixPath('/tmp/tmpodfbqyhx/b.txt'),
 PosixPath('/tmp/tmpodfbqyhx/c/d.txt')]

$ # This is Python 3.11.0a6 built from source
$ ~/python311a6/bin/python3.11 pathlib_rglob.py 
[PosixPath('/tmp/tmpobellcrw/a.txt'),
 PosixPath('/tmp/tmpobellcrw/b.txt'),
 PosixPath('/tmp/tmpobellcrw/c/d.txt')]
[PosixPath('/tmp/tmpobellcrw/a.txt'),
 PosixPath('/tmp/tmpobellcrw/b.txt'),
 PosixPath('/tmp/tmpobellcrw/c/d.txt')]

---

After much debugging I have tracked the issue to #32029
With that change, fnmatch.translate("*.*") gives (?s:(?>.*?\.).*)\Z.
With the change reverted (or on 3.11.0a6) its (?s:(?=(?P<g0>.*?\.))(?P=g0).*)\Z.
I don't know what the difference between those amounts to, but the former doesn't work while the latter does:

>>> import re
>>> re.fullmatch('(?s:(?>.*?\\.).*)\\Z', "a.txt")
>>> re.fullmatch('(?s:(?=(?P<g0>.*?\\.))(?P=g0).*)\\Z', "a.txt")
<re.Match object; span=(0, 5), match='a.txt'>

Your environment

• CPython versions tested on: 3.10.4, 3.11.0a6, 3.11.0a7, built from git checkouts of the relevant tag.
• Operating system and architecture: Ubuntu 20.04 amd64

Reacted by Shantanu

Activity

1. domdfcoding

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Apr 16, 2022
2. domdfcoding commented on Apr 16, 2022

   

   domdfcoding

   on Apr 16, 2022

   ContributorAuthor

   More actions

   @tim-one as you made the original change in #32029 could you take a look at this?
3. AlexWaygood

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   3.11only security fixesonly security fixes

   on Apr 16, 2022
6. serhiy-storchaka commented on Apr 17, 2022

   

   serhiy-storchaka

   on Apr 17, 2022

   Member

   More actions

   Ah, I see.

   >>> re.match(r'(?s:(?>.*?\.).*)\Z', 'a.txt')
   <re.Match object; span=(0, 5), match='a.txt'>
   >>> re.fullmatch(r'(?s:(?>.*?\.).*)\Z', 'a.txt')
   

   I thought that issues with fullmatch() and atomic grouping were solved, but it is not so.

   Use the following tests in your PR:

   diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
   index 959582e2f1..a5aa1b358d 100644
   --- a/Lib/test/test_re.py
   +++ b/Lib/test/test_re.py
   @@ -2173,6 +2173,10 @@ def test_fullmatch_possessive_quantifiers(self):
            self.assertIsNone(re.fullmatch(r'a*+', 'ab'))
            self.assertIsNone(re.fullmatch(r'a?+', 'ab'))
            self.assertIsNone(re.fullmatch(r'a{1,3}+', 'ab'))
   +        self.assertTrue(re.fullmatch(r'a++b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'a*+b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'a?+b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'a{1,3}+b', 'ab'))
    
            self.assertTrue(re.fullmatch(r'(?:ab)++', 'ab'))
            self.assertTrue(re.fullmatch(r'(?:ab)*+', 'ab'))
   @@ -2182,6 +2186,10 @@ def test_fullmatch_possessive_quantifiers(self):
            self.assertIsNone(re.fullmatch(r'(?:ab)*+', 'abc'))
            self.assertIsNone(re.fullmatch(r'(?:ab)?+', 'abc'))
            self.assertIsNone(re.fullmatch(r'(?:ab){1,3}+', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?:ab)++c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?:ab)*+c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?:ab)?+c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?:ab){1,3}+c', 'abc'))
    
        def test_findall_possessive_quantifiers(self):
            self.assertEqual(re.findall(r'a++', 'aab'), ['aa'])
   @@ -2217,6 +2225,10 @@ def test_fullmatch_atomic_grouping(self):
            self.assertIsNone(re.fullmatch(r'(?>a*)', 'ab'))
            self.assertIsNone(re.fullmatch(r'(?>a?)', 'ab'))
            self.assertIsNone(re.fullmatch(r'(?>a{1,3})', 'ab'))
   +        self.assertTrue(re.fullmatch(r'(?>a+)b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'(?>a*)b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'(?>a?)b', 'ab'))
   +        self.assertTrue(re.fullmatch(r'(?>a{1,3})b', 'ab'))
    
            self.assertTrue(re.fullmatch(r'(?>(?:ab)+)', 'ab'))
            self.assertTrue(re.fullmatch(r'(?>(?:ab)*)', 'ab'))
   @@ -2226,6 +2238,10 @@ def test_fullmatch_atomic_grouping(self):
            self.assertIsNone(re.fullmatch(r'(?>(?:ab)*)', 'abc'))
            self.assertIsNone(re.fullmatch(r'(?>(?:ab)?)', 'abc'))
            self.assertIsNone(re.fullmatch(r'(?>(?:ab){1,3})', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?>(?:ab)+)c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?>(?:ab)*)c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?>(?:ab)?)c', 'abc'))
   +        self.assertTrue(re.fullmatch(r'(?>(?:ab){1,3})c', 'abc'))
    
        def test_findall_atomic_grouping(self):
            self.assertEqual(re.findall(r'(?>a+)', 'aab'), ['aa'])
7. AlexWaygood

   added

   topic-regex

   on Apr 17, 2022
8. serhiy-storchaka commented on Apr 17, 2022

   

   serhiy-storchaka

   on Apr 17, 2022

   Member

   More actions

   Make the same change in POSSESSIVE_REPEAT.
11. serhiy-storchaka

    added a commit that references this issue on Apr 19, 2022

    gh-91616: re module, fix .fullmatch() mismatch when using Atomic Grou…

    e4e8895
12. serhiy-storchaka

    closed this as completedon Apr 22, 2022

Issue actions