GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-102511: Add C implementation of `os.path.splitroot()` by nineteendo · Pull Request #118089 · python/cpython

Conversation

@nineteendo

@nineteendo nineteendo commented Apr 19, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Benchmark

ntpath.py

PS C:\Users\wanne\cpython> main\python -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"; speedup-os.path.splitroot\python -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"
200000 loops, best of 5: 1.14 usec per loop # before
500000 loops, best of 5: 433 nsec per loop # after
# -> 2.63x faster
  • Issue: Add C implementation of os.path.splitroot() #102511

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nineteendo
Add C implementation of ntpath.splitroot()
c397a23
@bedevere-app bedevere-app Bot mentioned this pull request Apr 19, 2024
Add C implementation of os.path.splitroot() #102511
Closed
@blurb-it
📜🤖 Added by blurb_it.
4bd9734
@nineteendo
nineteendo marked this pull request as ready for review April 19, 2024 12:03
@bedevere-app bedevere-app Bot added the awaiting review label Apr 19, 2024
nineteendo added 4 commits April 19, 2024 14:11
@nineteendo
Follow Pep 7
0177f70
@nineteendo
Merge branch 'speedup-os.path.splitroot' of https://github.com/ninete…
e9cdec5
…endo/cpython into speedup-os.path.splitroot
@nineteendo
Fix memory leak
bb816e9
@nineteendo
Add C implementation of posixpath.splitroot()
d00131a
@nineteendo nineteendo changed the title gh-102511: Add C implementation of ntpath.splitroot() gh-102511: Add C implementation of os.path.splitroot() Apr 20, 2024
nineteendo added 3 commits April 20, 2024 14:11
@nineteendo
Revert newlines
a4e5d15
@nineteendo
Use _Py_splitroot()
7e1433c
@nineteendo
Rename to _path_splitroot replacing old one
5491152
@nineteendo
nineteendo requested review from brettcannon, ericsnowcurrently, ncoghlan and warsaw as code owners April 20, 2024 15:02
erlend-aasland
erlend-aasland reviewed Apr 22, 2024
View reviewed changes
Comment thread Modules/posixmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

zooba
zooba reviewed Apr 22, 2024
View reviewed changes

@zooba zooba left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great looking change! As Erlend pointed out, a couple of robustness things to check, and there's a potential behaviour change that's worth validating and describing in Doc\whatsnew\3.13.rst and possibly in other parts of the documentation.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/fileutils.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/fileutils.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/fileutils.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/importlib/_bootstrap_external.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Core and Builtins/2024-04-19-08-50-48.gh-issue-102511.qDEB66.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/posixmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

nineteendo and others added 4 commits April 22, 2024 17:09
@nineteendo @zooba
Update Misc/NEWS.d/next/Core and Builtins/2024-04-19-08-50-48.gh-issu…
2244d3f
…e-102511.qDEB66.rst

Co-authored-by: Steve Dower <[email protected]>
@nineteendo @zooba
Rename old function to _path_splitanchor
ca0761a
Co-authored-by: Steve Dower <[email protected]>
@nineteendo
Fix header
e1f32e9
@nineteendo
Direct C call
34d4d90
nineteendo
nineteendo commented Apr 22, 2024
View reviewed changes
Comment thread Modules/posixmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo

nineteendo commented Apr 22, 2024

Copy link
Copy Markdown
Contributor Author

Oh, come on, why must argclinic raise an error here? That isn't useful:

ValueError: _path_splitroot_ex: embedded null character in p
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba

zooba commented Apr 22, 2024

Copy link
Copy Markdown
Member

_path_splitroot_ex: embedded null character in p

Oh right, I forgot about this behaviour of argument clinic 😞 Might be a good reason to go to unicode after all.

Alternatively, you could handle the ValueError in the Python wrapper, split at the null, and then append it back onto the last element? Strictly speaking we're in the "invalid path --> invalid output" place again, as we don't actually support embedded nulls in paths at all, but there's no reason we need to go ahead and raise the error this early.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nineteendo added 4 commits April 23, 2024 08:08
@nineteendo
Allow embedded null
30d613b
@nineteendo
Fix segmentation fault
f78bad0
@nineteendo
Fix redefinition
5635da5
@nineteendo
Python wrapper
6f62c1f
@nineteendo

nineteendo commented Apr 24, 2024

Copy link
Copy Markdown
Contributor Author

It might just be my laptop or because I'm comparing with 3.12. (there doesn't seem to be a way to have 2 builds at the same time)
Otherwise we need to check what's different from your implementation. (if you haven't lost it, of course)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo

nineteendo commented Apr 24, 2024

Copy link
Copy Markdown
Contributor Author

OK, how's the build going?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eryksun

eryksun commented Apr 24, 2024

Copy link
Copy Markdown
Contributor

It might just be my laptop or because I'm comparing with 3.12. (there doesn't seem to be a way to have 2 builds at the same time)

You can use worktrees. Anyway, I still have my implementation, but there's no need to delve further into the comparison because the reason is obvious. I was testing the builtin function directly, not a wrapper function that called os.fspath() before calling the builtin. My plan was to switch to using path_t internally instead of a wrapper, but I never implemented that. Here's the performance I get for your PR with and without the wrapper.

> .\python_work -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"
1000000 loops, best of 5: 270 nsec per loop

> .\python_work -m timeit -s "import nt" "nt._path_splitroot_ex('//server/share/spam/eggs')"
2000000 loops, best of 5: 153 nsec per loop

It's 1.76 times faster without the wrapper.

nineteendo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo

nineteendo commented Apr 24, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

You can use worktrees.

Could you explain how I can use them?

I was testing the builtin function directly, not a wrapper function

Yeah, that matches the performance when I used path_t before. But I switched back to using a Python wrapper because NULL bytes and surrogates weren't handled properly. The code was quite ugly too with having to convert the input path to bytes on Unix.

I tried to work around the first issue by modifying path_converter, but I couldn't get surrogates to work, so I abandoned that idea and went for a simpler implementation.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eryksun

eryksun commented Apr 24, 2024

Copy link
Copy Markdown
Contributor

Here's a brief overview of worktrees in the developer's guide:

https://devguide.python.org/getting-started/git-boot-camp/#git-worktree

I tried to work around the first issue by modifying path_converter, but I couldn't get surrogates to work, so I abandoned that idea and went for a simpler implementation.

In another issue, the implementation of path_t could be generalized to support fields to configure the converter to use wide regardless of platform, to allow null characters, to allow arbitrary length paths (e.g. no 32767 length limit on Windows), and a new field such as bytes_input to determine whether a path result has to be converted back to bytes. The option to always use a wide-character path is a generalization of the current behavior on Windows. Argument Clinic would be extended to support the new options. The implementations of _path_splitroot_ex(), _path_normpath(), and _path_abspath() (if adopted) would benefit, and also the _path_is*() helpers on Windows.

nineteendo and erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eryksun
eryksun reviewed Apr 24, 2024
View reviewed changes
Comment thread Modules/posixmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo @eryksun
Update Modules/posixmodule.c
6a74f15
Co-authored-by: Eryk Sun <[email protected]>
@nineteendo nineteendo mentioned this pull request Apr 25, 2024
Generalize path_t for C level optimizations #118263
Closed
@nineteendo

nineteendo commented Apr 25, 2024

Copy link
Copy Markdown
Contributor Author

Let's leave path_t for a followup pull request, such that it can be implemented without a Python wrapper.

eryksun reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eryksun
eryksun approved these changes Apr 25, 2024
View reviewed changes

@eryksun eryksun left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks, nineteendo.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@zooba
zooba merged commit 10bb90e into python:main Apr 25, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Apr 25, 2024
@zooba

zooba commented Apr 25, 2024

Copy link
Copy Markdown
Member

That's quorum, thanks @nineteendo for bearing with us, and congrats on landing the patch!

erlend-aasland reacted with thumbs up emoji nineteendo reacted with rocket emoji
All reactions
  • 👍 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo
nineteendo deleted the speedup-os.path.splitroot branch April 25, 2024 09:09
@bedevere-bot

bedevere-bot commented Apr 25, 2024

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x has failed when building commit 10bb90e.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1380/builds/87) and take a look at the build logs.
  4. Check if the failure is related to this commit (10bb90e) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1380/builds/87

Failed tests:

  • test_os

Failed subtests:

  • test_fpathconf - test.test_os.TestInvalidFD.test_fpathconf

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/46780DAB-3371-45CA-A1E6-5178548A2DAA/data/Containers/Bundle/Application/2E63ACC7-4F0C-40C2-B867-2CAE5EAFD03D/iOSTestbed.app/python/lib/python3.13/test/test_os.py", line 2369, in test_fpathconf
    self.check(os.pathconf, "PC_NAME_MAX")
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/46780DAB-3371-45CA-A1E6-5178548A2DAA/data/Containers/Bundle/Application/2E63ACC7-4F0C-40C2-B867-2CAE5EAFD03D/iOSTestbed.app/python/lib/python3.13/test/test_os.py", line 2293, in check
    f(os_helper.make_bad_fd(), *args, **kwargs)
    ~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unrecognized configuration name
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo

nineteendo commented Apr 25, 2024

Copy link
Copy Markdown
Contributor Author

Hmm, it didn't fail in the previous commit: https://buildbot.python.org/all/#/builders/1380/builds/86
Is this random?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba

zooba commented Apr 25, 2024

Copy link
Copy Markdown
Member

This seems unrelated, and if it's only iOS, then we can ignore it. That buildbot has only just started running, and will be flushing out random issues for a little while.

nineteendo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nineteendo

nineteendo commented May 22, 2024

Copy link
Copy Markdown
Contributor Author

Eryk, can we backport this to 3.12 to fix the bug with ntpath.normpath()?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nineteendo added a commit to nineteendo/cpython that referenced this pull request May 22, 2024
@nineteendo
pythongh-102511: Speed up os.path.splitroot() with native helpers (py…
6872edd
…thonGH-118089)
@bedevere-app

bedevere-app Bot commented May 22, 2024

Copy link
Copy Markdown

GH-119394 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gpshead

gpshead commented May 31, 2024

Copy link
Copy Markdown
Member

Please add a unittest coverage of both the Python and C versions so that we ensure their behavior is the same.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eryksun

eryksun commented May 31, 2024

Copy link
Copy Markdown
Contributor

@gpshead, they both get tested as long as tests run on at least one POSIX system and one Windows system. The Python fallback implementation is only defined if the C version can't be imported. Thus the fallback for ntpath.splitroot() is only defined on POSIX, and the fallback for posixpath.splitroot() is only defined on Windows.

nineteendo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@aminalaee aminalaee mentioned this pull request Jun 20, 2024
os.path.splitroot: Python 3.13 python/typeshed#12172
Closed
sobolevn
sobolevn reviewed Sep 15, 2024
View reviewed changes

@sobolevn sobolevn left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR also changes first parameter name. It is now:

  • p for python version
  • path for C version

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@hugovk

hugovk commented Sep 15, 2024

Copy link
Copy Markdown
Member

Is this intentional? Does it need addressing before 3.13.0 in two weeks?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 15, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

I would prefer to address this, because:

  • It can break a user's code that used: os.path.splitroot(p="...")
  • It changes the typeshed typings, basically it makes the first param implicitly pos-only, because you cannot be sure which name it has: p or path
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 15, 2024

Copy link
Copy Markdown
Member

I will open a PR in a moment.

nineteendo reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn sobolevn mentioned this pull request Sep 15, 2024
gh-102511: Change the os.path.splitroot param name from path back to p #124097
Merged
@gianghungtien gianghungtien mentioned this pull request Jul 24, 2026
gh-122143: Fix os.path.splitroot() and os.path.normpath() for undecodable bytes paths on Windows #154585
Open
@coderabbitai coderabbitai Bot mentioned this pull request Aug 6, 2026
bench, parity: widen six floored-denominator ratio gates; scope the platform-specific parity fixtures pyre-lang/pyre#1071
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sobolevn sobolevn sobolevn left review comments
@eryksun eryksun eryksun approved these changes
@barneygale barneygale barneygale approved these changes
@zooba zooba zooba approved these changes
@erlend-aasland erlend-aasland erlend-aasland approved these changes
@brettcannon brettcannon Awaiting requested review from brettcannon
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently
@ncoghlan ncoghlan Awaiting requested review from ncoghlan
@warsaw warsaw Awaiting requested review from warsaw

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@erlend-aasland erlend-aasland

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@nineteendo @zooba @barneygale @erlend-aasland @eryksun @bedevere-bot @gpshead @hugovk @sobolevn
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.