GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-101362: Optimise pathlib by deferring path normalisation by barneygale · Pull Request #101560 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Feb 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

PurePath now normalises and splits paths only when necessary, e.g. when .name or .parent is accessed. The result is cached. This speeds up path object construction by around 4x.

PurePath.__fspath__() now returns an unnormalised path, which should be transparent to filesystem APIs (else pathlib's normalisation is broken!). This extends the earlier performance improvement to most impure Path methods, and also speeds up p.joinpath('bar') and p / 'bar'. edit: will fix separately.

This also fixes GH-76846 and GH-85281 by unifying path constructors and adding an __init__() method. edit: will fix separately.

  • Issue: Optimize pathlib path construction #101362

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood and mdboom reacted with rocket emoji
All reactions
  • 🚀 2 reactions
@barneygale
pythonGH-101362 - Optimise pathlib by deferring path normalisation
b428069 
`PurePath` now normalises and splits paths only when necessary, e.g. when
`.name` or `.parent` is accessed. The result is cached. This speeds up path
object construction by around 4x.

`PurePath.__fspath__()` now returns an unnormalised path, which should be
transparent to filesystem APIs (else pathlib's normalisation is broken!).
This extends the earlier performance improvement to most impure `Path`
methods, and also speeds up pickling, `p.joinpath('bar')` and `p / 'bar'`.

This also fixes pythonGH-76846 and pythonGH-85281 by unifying path constructors and
adding an `__init__()` method.
@bedevere-bot bedevere-bot added the awaiting review label Feb 4, 2023
@bedevere-bot bedevere-bot mentioned this pull request Feb 4, 2023
Optimize pathlib path construction #101362
Closed
barneygale added 4 commits February 4, 2023 16:32
@barneygale
Restore str force-casting behaviour; reduce diff a little.
e49e719
@barneygale
Fix pathlib usage error in importlib
a931986
@barneygale
Improve initialiser performance.
38f70bf
@barneygale
Add NEWS blurb
10844a6
@barneygale

barneygale commented Feb 4, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Constructing path objects is up to 4x faster with one argument:

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath' 'PurePath("foo/bar")' 
1000000 loops, best of 5: 2.01 usec per loop  # before
1000000 loops, best of 5: 495 nsec per loop  # after

More than 2x faster with two arguments:

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath' 'PurePath("foo", "bar")' 
1000000 loops, best of 5: 2.28 usec per loop  # before
1000000 loops, best of 5: 1.02 usec per loop  # after

~~And ~25% faster when joining arguments:~~

[edit: no longer true! ]

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath; p = PurePath("foo")' 'p.joinpath("bar")' 
1000000 loops, best of 5: 1.66 usec per loop  # before
1000000 loops, best of 5: 1.3 usec per loop  # after

But it's 12% slower when the path needs normalization, as with str()

$ ./python -m timeit -n 1000000 -s 'from pathlib import PurePath' 'str(PurePath("foo/bar"))' 
1000000 loops, best of 5: 2.96 usec per loop  # before
1000000 loops, best of 5: 3.31 usec per loop  # after

And 25% slower when when walking directories (where pathlib keeps everything normalized):

[edit: resolved! see comment]

$ ./python -m timeit -n 20 -s 'from pathlib import Path' 'list(Path().rglob("*"))' 
20 loops, best of 5: 53.4 msec per loop  # before
20 loops, best of 5: 66.5 msec per loop  # after

But still faster for filesystem operations that don't require normalization:

[edit: no longer true! this can't be properly fixed until other stuff lands]

$ ./python -m timeit -n 100000 -s 'from pathlib import Path' 'Path("README.rst").read_text()' 
100000 loops, best of 5: 26.1 usec per loop  # before
100000 loops, best of 5: 21.2 usec per loop  # after

$ ./python -m timeit -n 100000 -s 'from pathlib import Path' 'Path("README.rst").exists()' 
100000 loops, best of 5: 5.45 usec per loop  # before
100000 loops, best of 5: 2.97 usec per loop  # after
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
barneygale marked this pull request as ready for review February 4, 2023 18:28
@barneygale
barneygale requested review from jaraco and warsaw as code owners February 4, 2023 18:28
@barneygale
barneygale requested a review from AlexWaygood February 4, 2023 18:30
@barneygale
Store '_fspath' as non-empty string.
d5231b6
@barneygale
barneygale marked this pull request as draft February 7, 2023 20:35
@barneygale

barneygale commented Feb 7, 2023

Copy link
Copy Markdown
Contributor Author

I've found a couple other small optimizations which are best tackled in other PRs, so I'm marking this PR as a 'draft' for now.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale added the topic-pathlib label Feb 7, 2023
barneygale added 2 commits March 6, 2023 02:00
@barneygale
Merge branch 'main' into optimize-pathlib-part-2b
76b2db7
@barneygale
Undo addition of __init__() and change to _from_parsed_parts()
cc2c711
@barneygale barneygale changed the title GH-101362 - Optimise pathlib by deferring path normalisation GH-101362: Optimise pathlib by deferring path normalisation Mar 6, 2023
@AlexWaygood AlexWaygood added the performance Performance or resource usage label Mar 6, 2023
@barneygale
Fix pickling of paths created via walking.
cbf0fcd
@barneygale

barneygale commented Mar 6, 2023

Copy link
Copy Markdown
Contributor Author

I've undone the change to _from_parsed_parts(), which has restored directory-walking performance:

$ ./python -m timeit -n 20 -s 'from pathlib import Path' 'list(Path().rglob("*"))' 
20 loops, best of 5: 146 msec per loop  # before
20 loops, best of 5: 152 msec per loop  # after

Still a tiny bit slower than pre-PR.

The rest of the speedups/slowdowns mentioned in my previous comment are still there.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
barneygale marked this pull request as ready for review March 6, 2023 02:33
@barneygale
Simplify patch slightly
eb7087f
@barneygale barneygale mentioned this pull request Mar 10, 2023
GH-78079: Fix UNC device path root normalization in pathlib #102003
Merged
barneygale added 3 commits March 11, 2023 21:55
@barneygale
Remove unused import
3b53e27
@barneygale
Merge branch 'main' into optimize-pathlib-part-2b
dec98f2
@barneygale
Fix dodgy merge, add comment about _from_parsed_parts()
d9a6080
@barneygale

barneygale commented Mar 11, 2023

Copy link
Copy Markdown
Contributor Author

The change to importlib is necessary because it's relying on a bug in pathlib's path normalization:

  • pathlib strips trailing slash #65238

I think I need to solve that issue first, so I'm going to mark this PR as a draft (again!)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale
barneygale marked this pull request as draft March 11, 2023 23:32
This was referenced Mar 12, 2023
GH-101362: Omit path anchor from pathlib.PurePath()._parts #102476
Merged
pathlib strips trailing slash #65238
Closed
barneygale added 2 commits March 17, 2023 15:43
@barneygale
Stop returning unnormalised path from __fspath__()
9650dca
@barneygale
Merge branch 'main' into optimize-pathlib-part-2b
af27476
@barneygale
barneygale marked this pull request as ready for review March 17, 2023 16:20
@barneygale barneygale mentioned this pull request Mar 17, 2023
Optimize pathlib.PurePath.__fspath__() #102783
Closed
@barneygale
barneygale marked this pull request as draft March 17, 2023 16:45
@barneygale
Undo change to joinpath(), fix news blurb.
97995bd
@barneygale

barneygale commented Mar 17, 2023

Copy link
Copy Markdown
Contributor Author

This PR has strayed too far from the original implementation. I'm going to abandon it. New PR here:

  • GH-76846, GH-85281: Call __new__() and __init__() on pathlib subclasses #102789
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@barneygale barneygale closed this Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jaraco jaraco Awaiting requested review from jaraco
@warsaw warsaw Awaiting requested review from warsaw
@AlexWaygood AlexWaygood Awaiting requested review from AlexWaygood

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review performance Performance or resource usage topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

pathlib.Path._from_parsed_parts should call cls.__new__(cls)

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@barneygale @bedevere-bot @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.