Description

barneygale

opened on Mar 17, 2023

Issue body actions

Feature or enhancement

Return an unnormalized path from pathlib.PurePath.__fspath__()

Pitch

Code like open(Path('./README.txt')) or Path('/home//barney').iterdir() shouldn't require us to normalize the path in pathlib (e.g. remove . segments, doubled slashes, etc), as OS APIs are perfectly happy with unnormalized paths.

We can improve the performance of most Path methods, and the effective performance of passing a Path object to any API that accepts os.PathLike, by skipping normalization in __fspath__().

Prerequisites

Pathlib must not normalize paths on construction:

• Optimize pathlib path construction #101362

Pathlib's normalization must not change the meaning of paths:

• pathlib strips trailing slash #65238
• [pathlib] issues with Windows device paths #78079

Previous discussion

• GH-101362: Optimise pathlib by deferring path normalisation #101560

Linked PRs

• [WIP] GH-102783: Speed up pathlib.PurePath.__fspath__() by returning raw path #112397

Activity

1. barneygale

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Mar 17, 2023
2. AlexWaygood

   added

   performancePerformance or resource usagePerformance or resource usage

   topic-pathlib

   on Mar 17, 2023
3. eryksun commented on Mar 17, 2023

   

   eryksun

   on Mar 17, 2023

   Contributor

   More actions

   On Windows, an exception should be made to normalize extended paths in __fspath__(). If an extended path isn't normalized, the open will fail. One needs to replace slashes with backslashes; collapse repeated slashes; and resolve "." and ".." components.

   Scripts have to be prepared to handle extended paths coming from os.path.realpath(), __file__, and attributes of the sys module such as executable, prefix, argv, and path. Extended paths can also come from extension modules that use API functions such as GetModuleFileNameW() (if an EXE/DLL was loaded using an extended path), GetFinalPathNameByHandleW(), and any of the PathCch*() functions that implements the flag PATHCCH_ALLOW_LONG_PATHS.

   For path construction, there's an open issue that proposes to modify ntpath.join() to fully normalize any drive path or UNC path via normpath(). For paths without a drive, it's proposed to replace slashes with backslashes and collapse repeated slashes. For example, this change would support getting a useful result from something like ntpath.join(filename, "../spam//eggs.py") given filename is an extended path such as "\\?\C:\foo\bar.py".
4. barneygale commented on Mar 17, 2023

   

   barneygale

   on Mar 17, 2023

   ContributorAuthor

   More actions

   On Windows, an exception should be made to normalize extended paths in __fspath__(). If an extended path isn't normalized, the open will fail. One needs to replace slashes with backslashes; collapse repeated slashes; and resolve "." and ".." components.

   I don't think pathlib's normalization should change the meaning of paths. It sounds like we should adjust the normalization routine to preserve forward slashes, repeated slashes, and "." components, for these kinds of path. ".." components are already be preserved I think.
5. eryksun commented on Mar 17, 2023

   

   eryksun

   on Mar 17, 2023

   Contributor

   More actions

   I don't think pathlib's normalization should change the meaning of paths. It sounds like we should adjust the normalization routine to preserve forward slashes, repeated slashes, and "." components, for these kinds of path. ".." components are already be preserved I think.

   In almost all cases, extended paths are used solely to access a long path. The need to get a literal path is rare, especially in regard to slashes and "." and ".." component names. Microsoft's filesystems will fail an open that has forward slashes or repeated backslashes. NTFS will fail an open that has "." or ".." component names. FAT filesystems allow creating "." and ".." component names as a file's long name, with some other random short name, but it's very dysfunctional.

   People assume that open(filename / "../spam//eggs.txt") will just work. But if filename is an extended path -- one that's injected completely outside of the control of the script -- then this open will fail if the path isn't normalized. IMO, no purity in regards to literal paths is worth this headache.

   Reacted by Barney Gale
6. pitrou commented on Apr 20, 2023

   

   pitrou

   on Apr 20, 2023

   Member

   More actions

   OS APIs are perfectly happy with unnormalized paths

   What happens if a path is mounted on e.g. a FUSE filesystem or a Samba share? Does the OS unnormalize before handing the path to the filesystem?
7. iritkatriel

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 23, 2023
8. barneygale

   added a commit that references this issue on Nov 25, 2023

   pythonGH-102783: Speed up `pathlib.PurePath.__fspath__()` by returnin…

   2085494
9. barneygale

   added this to pathlib issueson Dec 2, 2023
10. barneygale

    moved this to In Progress in pathlib issueson Dec 4, 2023
11. barneygale commented on Dec 19, 2023

    

    barneygale

    on Dec 19, 2023

    ContributorAuthor

    More actions

    Resolving as "can't do", because #65238 is resolved as "won't fix"
12. barneygale

    closed this as not plannedon Dec 19, 2023
13. github-project-automation

    moved this from In Progress to Done in pathlib issueson Dec 19, 2023

Issue actions