GeezerVison is showing github.com — Support for sharing state between pathlib subclasses · Issue #100479 · python/cpython
Support for sharing state between pathlib subclasses #100479DescriptionFeature or enhancement
This enhancement proposes that we allow state to be shared between related instances of subclasses of pathlib.PurePath and pathlib.Path.
Pitch
Now that #68320 is resolved, users can subclass pathlib.PurePath and pathlib.Path directly:
import pathlib
class MyPath(pathlib.Path):
def my_custom_method(self):
pass
etc = MyPath('/etc')
etc_hosts = etc / 'hosts'
etc_hosts.my_custom_method()However, some user implementations of classes - such as TarPath or S3Path - would require underlying tarfile.TarFile or botocore.Resource objects to be shared between path objects (etc and etc_hosts in the example above). Such sharing of resources is presently rather difficult, as there's no single instance method used to derive new path objects.
This feature request proposes that we add a new PurePath.makepath() method, which is called whenever one path object is derived from another, such as in joinpath(), iterdir(), etc. The default implementation of this method looks something like:
def makepath(self, *args):
return type(self)(*args)Users may redefine this method in a subclass, in conjunction with a customized initializer:
class SessionPath(pathlib.Path):
def __init__(self, *args, session_id):
super().__init__(*args)
self.session_id = session_id
def makepath(self, *args):
return type(self)(*args, session_id=self.session_id)
etc = SessionPath('/etc', session_id=42)
etc_hosts = etc / 'hosts'
print(etc_hosts.session_id) # 42I propose the name "makepath" for this method due to its close relationship with the existing "joinpath" method: a.joinpath(b) == a.makepath(a, b).
Performance
edit: this change has been taken care of elsewhere, and so implementing this feature request should no longer have much affect on performance.
This change will affect the performance of some pathlib operations, because it requires us to remove the _from_parsed_parts() constructor, which is an internal optimization used in cases where path parsing and normalization can be skipped (for example, in the parents sequence). I suggest that, within the standard library, pathlib is not a particularly performance-sensitive module -- few folks reach to pathlib for reason of speed. Within pathlib itself, the savings from optimizing these "pure" methods are usually drowned out by the I/O costs of "impure" methods. With the appeal of this feature in mind, I believe the performance cost is justified.
However, if the performance degradation is considered unacceptable, there's a possible alternative: add a normalize keyword argument to the path initializer and to makepath(). This would require some serious internal surgery to make work, and might be difficult to communicate to users, so at this stage it's not my preferred route forward.
Previous discussion
https://discuss.python.org/t/make-pathlib-extensible/3428/47 (and replies)
Linked PRs
Activitybarneygale commented on Dec 23, 2022 ContributorAuthor More actions This will be needed to implement zipfile.Path using pathlib. Observe that it already has a _next() method that works exactly like what I'm proposing here:
|
def _next(self, at): |
|
return self.__class__(self.root, at) |
AlexWaygood commented on Dec 24, 2022 Exactly how big would the performance regression be? Have you tried running some benchmarks? barneygale commented on Dec 24, 2022 ContributorAuthor More actions Not yet; I'll do this before I mark the PR as 'ready' and share the numbers. AlexWaygood commented on Dec 24, 2022 My instinct is to say that it would be a shame to make pathlib performance worse, since it's already got a reputation in some quarters for having surprisingly bad performance in some ways compared to os.path. But it's hard to evaluate without knowing how big the performance regression would be, and in what situations you'd hit the regression :)
few folks reach to pathlib for reason of speed.
In my opinion, we should be trying to fix this rather than exacerbating the problem :) barneygale commented on Dec 24, 2022 ContributorAuthor More actions Absolutely fair!
Do you know of particular cases of (or complaints about) pathlib being slow? I'm aware that recursive globbing in pathlib can be slow, but not much else. We're increasingly calling os.path functions, some of which have C implementations, which should improve performance a little.
I'm of course keen for pathlib to be as fast as possible, but I don't want to prevent the future addition of an AbstractPath class either. AlexWaygood commented on Dec 24, 2022
Do you know of particular cases of (or complaints about) pathlib being slow?
I have no idea how many of these complaints, if any, still apply to the current implementation of pathlib, nor how many of these are realistically resolvable :) barneygale commented on Dec 24, 2022 ContributorAuthor More actions Mega, thank you. I'll study these carefully and write benchmarks.
Some initial thoughts:
- The performance cost of constructing
Path objects (vs constructing strings) stands out. This would surely benefit from an optimization pass, though it's unlikely we'll ever approach str construction performance.
- The change to the
parents implementation in GH-100479: Add optional blueprint argument to pathlib.PurePath #100481 may improve performance for some of these - I'll check.
- A C implementation of
pathlib would be wonderful but I suspect quite difficult! C implementations of select os.path functions would also help.
- Caching
stat() results would break user code, but there's a possible alternative: we could introduce something like a status() method that returns a new object with exists, is_dir, etc, attributes. Users could call path.status() once and check its attributes, rather than calling path.is_dir(), path.is_symlink(), etc, each of which would trigger their own stat() call.
barneygale commented on Apr 5, 2023 ContributorAuthor More actions AlexWaygood commented on Apr 5, 2023 I'll try to take a look soon! barneygale commented on May 5, 2023 ContributorAuthor More actions
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|