GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` by barneygale · Pull Request #104141 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented May 3, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

pathlib.Path.glob() now suppresses all OSError exceptions, except those raised from calling is_dir() on the top-level path.

Previously, glob() suppressed ENOENT, ENOTDIR, EBADF and ELOOP errors and their Windows equivalents. PermissionError was also suppressed unless it occurred when calling is_dir() on the top-level path. However, the selector would abort prematurely if a PermissionError was raised, and so glob() could return incomplete results.

Rationale

Globbing is a shell feature, per the glob module docs:

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell

When implemented by shells, globbing matches as much as possible without aborting.

In fact, other Python functions that walk a directory tree usually suppress errors, e.g.:

  • glob.glob()
  • os.walk() and os.fwalk() (configurable)
  • pathlib.Path.walk() (configurable)

The rationale is the same: an error from one file/directory isn't fatal for the walk, and we can continue walking elsewhere.

Finally, as noted above, pathlib.Path.glob() already ignores the most common kinds of OSError. This shows some recognition of the above rationale, though incomplete.

  • Issue: [pathlib] Option so that OSError does not block glob in pathlib library #90208

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@barneygale
pythonGH-90208 - Suppress OSError exceptions from pathlib.Path.glob()
f884b3f 
`pathlib.Path.glob()` now suppresses all OSError exceptions, except
those raised from calling `is_dir()` on the top-level path.

Previously, `glob()` suppressed ENOENT, ENOTDIR, EBADF and ELOOP
errors and their Windows equivalents. PermissionError was also
suppressed unless it occurred when calling `is_dir()` on the
top-level path. However, the selector would abort prematurely
if a PermissionError was raised, and so `glob()` could return
incomplete results.
@barneygale barneygale added the topic-pathlib label May 3, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 3, 2023
[pathlib] Option so that OSError does not block glob in pathlib library #90208
Closed
@bedevere-bot bedevere-bot added awaiting core review labels May 3, 2023
@barneygale

barneygale commented May 3, 2023

Copy link
Copy Markdown
Contributor Author

FAO @AlexWaygood, who pointed out the messy exception handling and poor test coverage in #104103
AlexWaygood reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale changed the title GH-90208 - Suppress OSError exceptions from pathlib.Path.glob() GH-90208: Suppress OSError exceptions from pathlib.Path.glob() May 4, 2023
@barneygale barneygale mentioned this pull request May 6, 2023
Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
Closed
@AlexWaygood

AlexWaygood commented May 6, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

(I'm not 100% confident I understand all the subtleties here, so I'd rather leave this one for Eryk or Steve to review, if that's okay!)
barneygale reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
Merge branch 'main' into pythongh-90208-pathlib-glob-oserror-ignore
1b7cbb7
@barneygale

barneygale commented May 8, 2023

Copy link
Copy Markdown
Contributor Author

Marking as a draft because it would be good to land #104292 first.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
barneygale marked this pull request as draft May 8, 2023 14:44
@barneygale barneygale mentioned this pull request May 8, 2023
GH-77609: Support following symlinks in pathlib.Path.glob() #104176
Closed
@barneygale
Merge branch 'main' into pythongh-90208-pathlib-glob-oserror-ignore
a59709c
@barneygale
barneygale marked this pull request as ready for review May 10, 2023 17:21
barneygale added 2 commits May 10, 2023 18:24
@barneygale
Reduce diff
556c702
@barneygale
Merge branch 'main' into pythongh-90208-pathlib-glob-oserror-ignore
0054195
@barneygale

barneygale commented May 10, 2023

Copy link
Copy Markdown
Contributor Author

@JelleZijlstra would you be interested in reviewing this? Thank you!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
JelleZijlstra
JelleZijlstra approved these changes May 10, 2023
View reviewed changes

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale looks solid, but this is enough of a behavior change that I'd probably not backport.

Your description made me ask why we ignore only that specific set of exceptions (ENOENT, ENOTDIR, EBADF, ELOOP). It looks like that set is used throughout pathlib in places where it makes more sense than here, so there may not have been a strong intent to ignore only those specific errors for globs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
barneygale reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 10, 2023
@barneygale

barneygale commented May 11, 2023

Copy link
Copy Markdown
Contributor Author

Thanks! I agree on not backporting.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
barneygale merged commit 94f30c7 into python:main May 11, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 11, 2023
carljm added a commit to carljm/cpython that referenced this pull request May 11, 2023
@carljm
Merge branch 'main' into supermon
19b3d9e 
* main: (27 commits)
  pythongh-87849: fix SEND specialization family definition (pythonGH-104268)
  pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384)
  pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367)
  pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369)
  pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383)
  pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287)
  pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185)
  pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368)
  pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
  pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182)
  pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364)
  pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539)
  pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292)
  pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202)
  pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013)
  pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355)
  pythongh-103960: Dark mode: invert image brightness (python#103983)
  pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253)
  pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354)
  pythongh-101819: Harden _io init (python#104352)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 11, 2023
@carljm
Merge branch 'main' into fixloadsuperspec
4456dd6 
* main:
  pythongh-87849: fix SEND specialization family definition (pythonGH-104268)
  pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384)
  pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367)
  pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369)
  pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383)
  pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287)
  pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185)
  pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368)
  pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
@JelleZijlstra JelleZijlstra mentioned this pull request May 22, 2023
PPC64LE Fedora Stable Clang 3.x fails test_gdb #104736
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@barneygale @AlexWaygood @JelleZijlstra @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.