GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-112906: Fix performance regression in pathlib path initialisation by barneygale · Pull Request #112907 · python/cpython

Conversation

@barneygale

@barneygale barneygale commented Dec 9, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This was caused by 76929fd, specifically its use of super() and its packing/unpacking of *args.

  • Issue: Performance regression in pathlib path instantiation #112906

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@barneygale
pythonGH-112906: Fix performance regression in pathlib path initialis…
911df8d
…ation

This was caused by 76929fd, specifically its use of `super()` and its
packing/unpacking `*args`.
@barneygale barneygale added performance Performance or resource usage topic-pathlib labels Dec 9, 2023
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 9, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Dec 9, 2023
Performance regression in pathlib path instantiation #112906
Closed
@barneygale

barneygale commented Dec 9, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author
$ ./python -m timeit -n 1000000 -s "from pathlib import PurePath" "PurePath('a', 'b')"
1000000 loops, best of 5: 636 nsec per loop  # before regression (baseline)
1000000 loops, best of 5: 827 nsec per loop  # after regression (30% slower)
1000000 loops, best of 5: 662 nsec per loop  # after this fix (4% slower)

The last few % are hard to recoup unfortunately

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale barneygale added the skip news label Dec 9, 2023
@barneygale

barneygale commented Dec 9, 2023

Copy link
Copy Markdown
Contributor Author

Skipping news because this only regressed yesterday.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

terryjreedy
terryjreedy requested changes Dec 9, 2023
View reviewed changes

@terryjreedy terryjreedy left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that PurePath may be instantiated multiple times in an app, the speed recovery looks worthwhile. The net 5% loss is the cost of a new class.

The logic looks correct (and I presume testing verifies it); just add docstrings with minimal explanation. Without the explanation in the issue and PR, this could puzzle future readers.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

barneygale reacted with heart emoji
All reactions
  • ❤️ 1 reaction
Comment thread Lib/pathlib/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/pathlib/_abc.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added awaiting changes and removed awaiting core review labels Dec 9, 2023
@bedevere-app

bedevere-app Bot commented Dec 9, 2023

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
Docstrings
4c327d7
@barneygale

barneygale commented Dec 9, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood
AlexWaygood reviewed Dec 9, 2023
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered either of these two alternatives? (Both diffs are relative to main, not this PR branch)

diff --git a/Lib/pathlib/__init__.py b/Lib/pathlib/__init__.py
index f4668ab327..626f817733 100644
--- a/Lib/pathlib/__init__.py
+++ b/Lib/pathlib/__init__.py
@@ -90,7 +90,9 @@ def __init__(self, *args):
                         "object where __fspath__ returns a str, "
                         f"not {type(path).__name__!r}")
                 paths.append(path)
-        super().__init__(*paths)
+        # As an optimisation, avoid calling super().__init__() here
+        self._raw_paths = paths
+        self._resolving = False

or

diff --git a/Lib/pathlib/__init__.py b/Lib/pathlib/__init__.py
index f4668ab327..5c605547c0 100644
--- a/Lib/pathlib/__init__.py
+++ b/Lib/pathlib/__init__.py
@@ -90,7 +90,7 @@ def __init__(self, *args):
                         "object where __fspath__ returns a str, "
                         f"not {type(path).__name__!r}")
                 paths.append(path)
-        super().__init__(*paths)
+        super().__init__(paths)

     def __reduce__(self):
         # Using the parts tuple helps share interned path parts
diff --git a/Lib/pathlib/_abc.py b/Lib/pathlib/_abc.py
index 4808d0e61f..2d7fba392a 100644
--- a/Lib/pathlib/_abc.py
+++ b/Lib/pathlib/_abc.py
@@ -206,7 +206,7 @@ class PurePathBase:
     )
     pathmod = os.path

-    def __init__(self, *paths):
+    def __init__(self, paths):
         self._raw_paths = paths
         self._resolving = False

The first is slightly faster than your PR here; the second is slightly slower.

These both feel like slightly simpler patches. The second might also provide speedups to third-party subclasses of PurePathBase as well as subclasses in the Lib/pathlib/ directory

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

barneygale reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@barneygale

barneygale commented Dec 9, 2023

Copy link
Copy Markdown
Contributor Author

Thanks Alex

First alternative: it's been a while since I've dealt with this, but I think not calling super() from __init__() can snooker users of multiple inheritance in certain cases. Maybe? My editor warns me about it but I don't recall specifics.

Second alternative: would make the initialiser signature of PurePathBase and PathBase different from PurePath and Path, which I think would be hard to explain to users.

AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale

barneygale commented Dec 9, 2023

Copy link
Copy Markdown
Contributor Author

I think I was wrong about the first alternative: it's fine to omit the super() call. Will adjust the patch.

AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@barneygale
Greatly simplify patch (h/t Alex Waygood)
7136b32
@AlexWaygood

AlexWaygood commented Dec 9, 2023

Copy link
Copy Markdown
Member

Yeah, if you're looking to do cooperative multiple inheritance, it's always advisable to always call super().__init__() when overriding __init__ in a subclass.

Outside of that situation, though, I don't think there's any reason to say you have to always call super().__init__() in a subclass -- there are often good reasons to avoid doing so. Linters often warn about it because a common mistake is to accidentally forget to include the super().__init__() call in the subclass -- but we know what we're doing here, so it's all good :)

The cooperative multiple inheritance concern doesn't apply here either, I don't think, since PurePath itself only has one base -- PurePathBase. But it's also been a while since I've tried to do cooperative multiple inheritance with anything; I might have to rewatch https://www.youtube.com/watch?v=EiOglTERPEo&list=PLd9YubueMBAAHenyO2yyf4bg27uwzx-Fc&index=11 to brush up :)

Second alternative: would make the initialiser signature of PurePathBase and PathBase different from PurePath and Path, which I think would be hard to explain to users.

Yeah, good point!

barneygale reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood
AlexWaygood reviewed Dec 9, 2023
View reviewed changes
Comment thread Lib/pathlib/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@barneygale @AlexWaygood
Update Lib/pathlib/__init__.py
3dbb483
Co-authored-by: Alex Waygood <[email protected]>
AlexWaygood
AlexWaygood approved these changes Dec 9, 2023
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting changes labels Dec 9, 2023
@barneygale
barneygale enabled auto-merge (squash) December 9, 2023 23:49
@barneygale
barneygale merged commit 23df46a into python:main Dec 10, 2023
@bedevere-app bedevere-app Bot removed the awaiting merge label Dec 10, 2023
@barneygale

barneygale commented Dec 10, 2023

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews, both!

AlexWaygood reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@barneygale @AlexWaygood @aisk
pythonGH-112906: Fix performance regression in pathlib path initialis…
0f4cd11
…ation (python#112907)

This was caused by 76929fd, specifically its use of `super()` and its
packing/unpacking `*args`.

Co-authored-by: Alex Waygood <[email protected]>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@barneygale @AlexWaygood @Glyphack
pythonGH-112906: Fix performance regression in pathlib path initialis…
426d021
…ation (python#112907)

This was caused by 76929fd, specifically its use of `super()` and its
packing/unpacking `*args`.

Co-authored-by: Alex Waygood <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@terryjreedy terryjreedy terryjreedy requested changes
@AlexWaygood AlexWaygood AlexWaygood approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage skip news topic-pathlib

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@barneygale @AlexWaygood @terryjreedy
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.