GeezerVison is showing github.com — Optimize pathlib path construction · Issue #101362 · python/cpython
Optimize pathlib path construction #101362DescriptionPathlib is slow. One of the most obvious symptoms is that pathlib.PurePath objects are slow to construct. We should be able to speed construction up without making other parts of pathlib slower.
Two possible approaches:
- Optimize the existing machinary of path construction:
__new__(), _from_parts(), _parse_parts(), _parse_args().
- Perform less work in the constructor: defer parsing, joining and normalization until needed.
Linked PRs
Reacted by Alex Waygood, sobolevn, Jacob Nilsson, ProgramRipper, sigma67, Gabriele N. Tornetta and tblazejczyk-qblox Activitybarneygale commented on Jan 29, 2023 ContributorAuthor More actions I'd like to land #101363 before I put the first PR up for this issue. zooba commented on Feb 22, 2023 Does the PR cope with WindowsPurePath and PosixPurePath and converting between them? barneygale commented on Feb 22, 2023 ContributorAuthor More actions Could you clarify? The PRs maintain the behaviour that attempting to instantiate pathlib.PurePath will give you a PureWindowsPath or a PurePosixPath depending on your platform, but pathlib doesn't support converting between Windows and POSIX paths as I understand it. zooba commented on Feb 24, 2023 This behaviour should be preserved:
Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> PurePosixPath(p)
PurePosixPath('a/b/c')
>>> PurePath(p)
PureWindowsPath('a/b/c')
>>> PosixPath(p)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__3847v3x7pw1km\lib\pathlib.py", line 962, in __new__
raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'PosixPath' on your system
>>> WindowsPath(p)
WindowsPath('a/b/c')
>>>
barneygale commented on Feb 24, 2023 ContributorAuthor More actions Right! That will be broken by #101667 as things stand:
>>> from os import fspath
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> fspath(p)
'a\\b\\c'
>>> PurePosixPath(fspath(p))
PurePosixPath('a\\b\\c')
>>> PurePosixPath(p)
PurePosixPath('a\\b\\c')It doesn't appear to be documented or tested behaviour, and it feels odd to me that PurePosixPath(p) can give a different result to PurePosixPath(fspath(p)). I'll try to find a good way forwards! barneygale commented on Feb 24, 2023 ContributorAuthor More actions This feature doesn't really work with drives or roots:
>>> PurePosixPath(PureWindowsPath('//server/share/dir'))
PurePosixPath('\\\\server\\share\\/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir'))
PurePosixPath('c:\\/dir')
>>> PurePosixPath(PureWindowsPath('/dir'))
PurePosixPath('\\/dir')As far as I can tell, no one has ever logged a bug about it.
However, using PurePosixPath(PureWindowsPath(...).as_posix()) works everywhere:
>>> PurePosixPath(PureWindowsPath('//server/share/dir').as_posix())
PurePosixPath('//server/share/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir').as_posix())
PurePosixPath('c:/dir')
>>> PurePosixPath(PureWindowsPath('/dir').as_posix())
PurePosixPath('/dir')So I'm tempted to conclude that converting with PurePosixPath(PureWindowsPath(...)) alone is not supported, and that it only happened to work in some limited circumstances. Users can add .as_posix() to signal their intent and make it work reliably. What do you reckon? zooba commented on Feb 24, 2023 I think the direct conversion should either be consistent with fspath(p) or p.parts, but no strong preference which. As you say, neither makes a huge amount of sense, and the workaround (.as_posix()) has been there for a long time.
Might be worth adding a short note to the docs warning that the constructor can't reliably convert from different pathlib objects, and maybe specify whether it will fspath or .parts them.
On another perf note, is it possible that parsing the path up front isn't necessary? Obviously it'll save the most time to keep a single string literal around and parse it later, but I don't personally have a good feel for whether that's common or not. (Obviously if it's available pre-parsed then keep it.) barneygale commented on Mar 22, 2023 ContributorAuthor More actions For anyone following along, I think this ticket can be resolved if/when this PR lands:
barneygale commented on Apr 9, 2023 ContributorAuthor More actions Resolving this issue. It's now much cheaper to construct pathlib.PurePath and Path objects. Some operations like str() and joinpath() are also cheaper.
Future work:
Reacted by Alex Waygood and skshetry
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|