GeezerVison is showing github.com — pathlib.Path.iterdir doesn't raise an exception until you start iterating · Issue #78722 · python/cpython
pathlib.Path.iterdir doesn't raise an exception until you start iterating #78722Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-08-29.08:20:21.120>
labels = ['3.8', 'type-feature', 'library']
title = "pathlib.Path.iterdir doesn't throw an exception until you start iterating"
updated_at = <Date 2019-05-23.22:47:36.119>
user = 'https://bugs.python.org/PaulPinterits' bugs.python.org fields:
activity = <Date 2019-05-23.22:47:36.119>
actor = 'cheryl.sabella'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-08-29.08:20:21.120>
creator = 'Paul Pinterits'
dependencies = []
files = []
hgrepos = []
issue_num = 34541
keywords = ['patch']
message_count = 3.0
messages = ['324307', '324332', '324334']
nosy_count = 3.0
nosy_names = ['pitrou', 'Paul Pinterits', 'prudvinit']
pr_nums = ['8996', '8999']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34541'
versions = ['Python 3.8'] Linked PRs
ActivityPaulPinterits commented on Aug 29, 2018 MannequinAuthor More actions The fact that Path.iterdir() only throws exceptions once you start iterating over it makes it very difficult to write correct code.
Let's look at an example: We'll iterate over all children of a directory and print their file size.
If we try to do it like this, the try...except has no effect whatsoever:
try:
children = path_that_doesnt_exist.iterdir()
except FileNotFoundError:
print("directory doesn't exist")
for path in children:
print(path.stat().st_size)If we explicitly check whether the path exists (and is a directory), we end up with a race condition:
if path_that_doesnt_exist.is_dir():
print("directory doesn't exist")
else:
for path in children:
print(path.stat().st_size)We can wrap the whole loop in a try...except, but then we might end up catching exceptions we didn't intend to catch. (For example, the exception that's thrown when we try to get the size of a broken symlink.)
try:
for path in path_that_doesnt_exist.iterdir():
print(path.stat().st_size) # this can also throw FileNotFoundError
except FileNotFoundError:
print("directory doesn't exist")We can manually call next on the iterator inside of a try..except block, but that's awfully verbose and requires knowledge about iterators and the next function:
children = iter(path_that_doesnt_exist.iterdir())
while True:
try:
path = next(children)
except FileNotFoundError:
print("directory doesn't exist")
break
print(path.stat().st_size)Or we can turn the iterator into a list inside of a try...except, which seems to be the best option, but completely defeats the point of having an iterator:
try:
children = list(path_that_doesnt_exist.iterdir())
except FileNotFoundError:
print("directory doesn't exist")
else:
for path in children:
print(path.stat().st_size)As you can see, writing correct (and good) code with iterdir is more difficult than it has any right to be. Please change this behavior so that exceptions are thrown immediately when iterdir is called. PaulPinterits commented on Aug 29, 2018 MannequinAuthor More actions As an afterthought, I'd like to suggest an alternative solution: If changing the iterdir behavior is not possible or not desirable for some reason, please add a Path.listdir method that returns a list instead of an iterator. Lazy file system operations can often introduce unnecessary race conditions in the code, so I believe it would be very useful if every lazy method had an eager equivalent. prudvinit commented on Aug 29, 2018 Made changes, pathlib.Path('.').iterdir() now throws a FileNotFoundError if the path is not valid. barneygale commented on Apr 13, 2022 Work-around: call list(path.iterdir()):
try:
children = list(path_that_doesnt_exist.iterdir())
except FileNotFoundError:
print("directory doesn't exist")
for path in children:
print(path.stat().st_size)I don't think we can change the iterdir() behaviour at this stage, nor does it justify the introduction of a similar method like listdir() to my mind. Reacted by C.A.M. Gerlach CAM-Gerlach commented on Jan 8, 2023 Yeah, the fact that it returns a generator/lazy iterator seems pretty fundamental to iterdir's behavior, and matches the other current and past iter* functions/methods in Pythons, as well as pathlib.glob and similar methods (as well as builtins like range and others) which would make it suddenly behaving eagerly quite surprising to many/most users, as well as sacrificing performance/efficiency.
list(<lazy iterator>) to make something iterable is a common idiom in Python to handle this situation (and others), and adding a separate method to do the same thing eagerly seems thoroughly redundant.
barneygale commented on Apr 14, 2023 I'm going to close this issue per CAM's rationale above. Use list(path.iterdir()) if you need to handle exceptions before you iterate. Thanks all. Reacted by C.A.M. Gerlach jaraco commented on Jul 18, 2023 I don't believe list(iterdir()) is a suitable workaround, as it requires the entire directory to be materialized in memory, increasing storage with little benefit. I'd argue that itertools (or similar) should provide a mechanism to filter or stop on exceptions, so consuming the iterable can continue to be lazy.
more_itertools.iter_except almost handles this case, but (a) requires the callable to be passed and (b) can't handle StopIteration as a stop condition. barneygale commented on Jul 26, 2023 Upon reflection I agree. I've put a PR up. Do you have a preference on whether it should be backported? changed the title [-]pathlib.Path.iterdir doesn't throw an exception until you start iterating[/-] [+]pathlib.Path.iterdir doesn't raise an exception until you start iterating[/+] on Aug 28, 2023barneygale commented on Sep 2, 2023 added a commit that references this issue on Sep 2, 2023GH-78722: Raise exceptions from `pathlib.Path.iterdir()` without dela… barneygale commented on Sep 2, 2023 jaraco commented on Sep 3, 2023 Thanks for the fix!
Upon reflection I agree. I've put a PR up. Do you have a preference on whether it should be backported?
This feels like a change in behavior / improvement, so probably not suitable for a backport.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|