GeezerVison is showing github.com — src: optimize NormalizeString to cut allocations and copies by whoekage · Pull Request #63753 · nodejs/node
Conversation
What this changes
NormalizeString() in src/path.cc is the lexical path-normalization core
behind PathResolve(), NormalizeFileURLOrPath() and ToNamespacedPath().
It collapses //, removes ., resolves .., and strips trailing
separators, and it runs during module and package resolution, the compile
cache, --permission fs checks, and (on Windows) on every fs syscall via
ToNamespacedPath().
This rewrites the .. handling to a write-position rewind with a dotdot
floor index, the approach used by Go's path/filepath.Clean, and switches
segment building to in-place appends.
Before
For every .. segment the old code:
- ran
res = res.substr(0, idx), which allocated a new string and copied the
entire surviving prefix (O(n²) copy volume on deep-backtrack paths);
- ran two
find_last_of(separator) scans, one to truncate and one to
recompute lastSegmentLength.
For every normal segment it built two or three temporary std::strings
(std::string(separator) + std::string(path.substr(...))), and res was
never reserve()d.
After
- A
dotdot floor index marks where backtracking must stop. Resolving ..
rewinds the write position to the previous separator and calls
res.resize(), so there is no allocation, no prefix copy and no
find_last_of.
- Segments are written with
push_back() and append(ptr, count), with no
temporary strings.
res.reserve(path.size()) is called once (the output is always no longer
than the input).
lastSegmentLength and the "does res already end with .." check are
gone, subsumed by dotdot.
The change is 30 insertions and 33 deletions in src/path.cc.
Benchmarks
Measured with an A/B harness that compiles the old and new implementations
into a single binary (clang -O2, Apple M-series), corroborated by a cctest
timing run. Time per call:
| input |
before |
after |
speedup |
/usr/local/bin/node (typical) |
71 ns |
37 ns |
1.9x |
| deep backtrack |
222 ns |
94 ns |
2.4x |
| 200 segments, no backtrack |
3981 ns |
1647 ns |
2.4x |
50x a/ + 50x ../ |
2410 ns |
522 ns |
4.6x |
Heap allocations per call, counted with an operator new override: the
50x a/ + 50x ../ case drops from 41 to 1, the 200-segment path from 6 to 1,
and typical or short paths stay at 0 (small-string optimization).
Correctness
The output is byte for byte identical to the previous implementation. I
verified this with a differential test of 1.2M random inputs (length 0 to 40
over {'/','\\','.','a','b','c'}, both allowAboveRoot values) against a
frozen copy of the old implementation, under both POSIX semantics (separator
/) and Windows semantics (IsPathSeparator matching / and \, with
separators / and \): 0 mismatches. The existing cctest PathResolve
cases pass, and new cases lock the touched behaviors (deep backtrack to root,
the .. floor with allowAboveRoot, and ., // and trailing-separator
collapse).
Scope
This is a hot helper, but on POSIX it is a small fraction of end-to-end JS
workloads. The main beneficiaries are path-resolution-heavy code and, on
Windows, fs operations, where ToNamespacedPath runs on every syscall. The
change is purely a constant-factor and allocation improvement with identical
observable behavior.
Prior art: Go path/filepath.Clean (lazybuf plus a dotdot index). A
related allocation optimization in this subsystem that was accepted is #50288.
Checklist
-
make -j4 test (UNIX) passes for the touched area (cctest PathTest*)
- tests are included
- documentation is changed or added
- commit message follows commit guidelines
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
The changes look good overall. Since this is oriented towards improving performance, could you add some benchmarks here so that we can verify the improvements here? It would also be helpful to have those for any potential future changes.
|
whoekage reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This comment was marked as resolved.
This comment was marked as resolved.
Rewrite the parent-dir ("..") handling in NormalizeString() to use a
write-position rewind with a "dotdot" floor index, the approach used by
Go's path/filepath.Clean, and build segments with in-place appends
instead of temporary-string concatenation.
Previously each ".." did `res = res.substr(0, idx)`, which allocated a
new string and copied the whole surviving prefix (O(n^2) copy volume on
deep-backtrack paths), plus two find_last_of() scans. Each normal
segment built two or three temporary std::strings, and res was never
reserve()d.
Now ".." rewinds the write position to the previous separator and
resize()s, so there is no allocation, no prefix copy and no
find_last_of call. Segments are written with push_back() and append().
The output is byte for byte identical to the previous implementation.
Signed-off-by: Phail' Sharkaev <[email protected]>
process.permission.has() resolves the reference natively through
PathResolve() and NormalizeString() in src/path.cc before the granted
tree lookup, and string references are passed through from JS unchanged,
so it is a JS-reachable entry point for the native path normalization on
all platforms.
The new benchmark feeds it paths of different shapes: already
normalized, `.` segments with repeated separators, a few `..` segments,
many segments without backtracking, and a deep descent followed by an
equally deep backtrack. This makes changes to NormalizeString()
measurable with benchmark/compare.js.
Signed-off-by: Phail' Sharkaev <[email protected]>
|
@StefanStojanovic added benchmark/permission/permission-processhas-fs-read-paths.js. NormalizeString() has no direct JS binding; process.permission.has('fs.read', p) resolves p natively via PathResolve() with no JS-side normalization, on all platforms, so it is used as the entry point. Five path shapes are covered.
benchmark/compare.js --analyze, 30 runs each, both binaries built from the same tree (old = src/path.cc from main). Improvement is the ops/sec change new vs old, CI is the half-width of the confidence interval as % of old, stars
are Welch t-test significance (* p<0.05, *** p<0.001):
| pathType |
path shape |
improvement |
±95% CI |
significance |
normalized |
<root>/benchmark/permission/valid-file |
+0.23 % |
2.82 % |
none |
dot-segments |
<root>/./benchmark//./permission/./valid-file |
+0.73 % |
2.88 % |
none |
dotdot |
<root>/a/b/c/d/e/../../../../../valid-file |
+3.51 % |
2.89 % |
* |
deep |
<root> + 200 × /segment |
+20.61 % |
2.81 % |
*** |
dotdot-heavy |
<root> + 50 × /a + 50 × /.. |
+17.69 % |
3.00 % |
*** |
For reference, NormalizeString() alone on the same inputs (standalone, clang -O2, operator new counter): heap allocations per call drop from 2/2/8/7/54 to 1 for the five shapes respectively, and time per call from 199/204/510/4994/2816 ns to 99/104/123/2250/528 ns. process.permission.has() costs ~11 µs per call in total, so for short paths the saving is under 1 % and within noise; the predicted savings for dotdot, deep and dotdot-heavy match the measured improvements. deep and dotdot-heavy remain significant after Holm-Bonferroni correction for the 5 comparisons.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
codecov
Bot
commented
Sep 7, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.19%. Comparing base (695f01f) to head (6a2778d).
⚠️ Report is 896 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #63753 +/- ##
==========================================
- Coverage 90.19% 90.19% -0.01%
==========================================
Files 771 771
Lines 264628 264883 +255
Branches 50237 50311 +74
==========================================
+ Hits 238683 238899 +216
- Misses 16953 16962 +9
- Partials 8992 9022 +30
... and 38 files with indirect coverage changes
🚀 New features to boost your workflow:
- ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Labels
c++
Issues and PRs that require attention from people who are familiar with C++.
needs-ci
PRs that need a full CI run.
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|