GeezerVison is showing github.com — Pathlib.iterdir semantics change dramatically under Python 3.13 · Issue #129871 · python/cpython
Pathlib.iterdir semantics change dramatically under Python 3.13 #129871DescriptionBug report
Bug description:
This code behaves very differently on Python 3.13 than 3.12:
(p for p in pathlib.Path('does-not-exist').iterdir()) 🐚 py -3.12 -c "import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())"
🐚 py -3.13 -c "import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())"
Traceback (most recent call last):
File "<string>", line 1, in <module>
import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pathlib/_local.py", line 575, in iterdir
with os.scandir(root_dir) as scandir_it:
~~~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'In Python 3.12, the generator expression was evaluated lazily, not invoking path.iterdir() until the generator was consumed. On 3.13, at least a portion of the generator is evaluated, triggering the exception when the dir does not exist.
I discovered this issue in jaraco/jaraco.develop#26.
I looked at What's New for Python 3.13, and there's only one mention of generator expressions regarding mutation of locals in generator expressions, which doesn't seem to be relevant here.
That change mentions #74929, so maybe that change is also implicated in the change in execution order.
CPython versions tested on:
3.13
Operating systems tested on:
macOS Reacted by Christophe Bornet Activityjaraco commented on Feb 8, 2025 Silly me. I just realized this issue probably isn't about a change to the generator expression but a change to pathlib. changed the title [-]Generator semantics change dramatically under Python 3.13[/-] [+]Pathlib.iterdir semantics change dramatically under Python 3.13[/+] on Feb 8, 2025jaraco commented on Feb 8, 2025 Indeed, the generator expression is not relevant:
🐚 py -3.12 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()"
🐚 py -3.13 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
import pathlib; pathlib.Path('does-not-exist').iterdir()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pathlib/_local.py", line 575, in iterdir
with os.scandir(root_dir) as scandir_it:
~~~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'
tomasr8 commented on Feb 8, 2025 Bisected to #107320 so confirming that it's related to iterdir() and not the generator itself. Reacted by Jason R. Coombs jaraco commented on Feb 8, 2025 I was even involved in that issue and had completely forgotten.
So it was an intentional change, even though it doesn't appear in the "what's new". At the very least, we should add a note to "what's new", as this behavior might be surprising.
I'm also thinking we should revisit our assumptions from #78722 in light of this new experience. It demonstrates that the change in behavior isn't necessarily beneficial and there may be other use-cases out there that are relying on the late evaluation of the error condition.
I don't feel strongly about it, and I'm happy to proceed with this new behavior, but we should ask ourselves if a rollback is warranted to retain consistency across Python versions. barneygale commented on Feb 9, 2025 I'll get a PR up for adding it to the whatsnew, and perhaps to the Path.iterdir() docs
I still prefer the new behaviour - it meets most users expectations better, it makes exception handling much more straightforward, and it matches how os.scandir() raises exceptions too. Reacted by Jason R. Coombs and Petr Viktorin cbornet commented on Feb 20, 2025 This is indeed a big change.
Some libs such as anyio are relying on the fact that iterdir() doesn't do blocking syscalls and defer to a thread only when it's iterated. barneygale commented on Feb 21, 2025 Maybe we should revert the iterdir() change then, despite it being a little awkward to handle exceptions previously. jaraco commented on Feb 27, 2025 I'm so conflicted on this, I can't even advocate for one approach over the other. I'm inclined to say we should raise the issue with a larger group. I do think it would be worthwhile to make a decision quickly to reduce the duration of exposure. Maybe just a post in Core Dev discuss? I'm happy to defer to your judgment Barney, but let me know if you'd like my help garnering a wider consensus. nh2 commented on Jun 28, 2025 emmatyping commented on Jun 28, 2025 I think I would expect an iterator to lazily surface exceptions as it does work. If a user wants to surface them sooner they can always wrap the result in a tuple or list themselves to drive the iterator.
Given that this change broke anyio, I think it should be reverted. Since there is a not-too-complicated alternative (wrap Path.iterdir() in list() if you want eager exceptions to be raised), the motivation does not seem to justify breaking behavior of downstream users to me. serhiy-storchaka commented on Jun 29, 2025 I think that raising an exception for non-existing directory (and for non-directory) is right. If you want to make this exception lazy, just wrap the creation of the iterator in a generator function:
def path_iterdir(p):
yield from p.iterdir()
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|