GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fs: improve `ExistsSync` performance on Windows by anonrig · Pull Request #53537 · nodejs/node

Conversation

@anonrig

@anonrig anonrig commented Jun 21, 2024

Copy link
Copy Markdown
Member

Simplifies code by using std::filesystem. Since std::filesystem is C++17, this pull-request can be backported to 18 and 20.

The following benchmark shows 2% improvement, but I'm sure if we had a Windows benchmark-ci machine it would have better results. Especially for symlinks, we are now not executing 2 different operating-system calls, and only one.

FYI: std::filesystem::exists follows symlinks.

Benchmark code:

const fs = require('node:fs');
const exists = fs.existsSync(__filename);
console.log(exists);

Benchmark result:

PS C:\Users\yagiz\Desktop\coding\node> hyperfine ".\out\Release\node .\exists-bench.js" " .\node-pr .\exists-bench.js" --warmup 10
Benchmark 1: .\out\Release\node .\exists-bench.js
  Time (mean ± σ):      48.8 ms ±   2.9 ms    [User: 2.2 ms, System: 3.3 ms]
  Range (min … max):    41.6 ms …  55.3 ms    56 runs

Benchmark 2:  .\node-pr .\exists-bench.js
  Time (mean ± σ):      47.7 ms ±   2.0 ms    [User: 1.3 ms, System: 2.8 ms]
  Range (min … max):    42.2 ms …  50.5 ms    60 runs

Summary
   .\node-pr .\exists-bench.js ran
    1.02 ± 0.07 times faster than .\out\Release\node .\exists-bench.js

cc @nodejs/fs @nodejs/platform-windows

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bricss, kermanx, and ibodev1 reacted with rocket emoji
All reactions
  • 🚀 3 reactions
@anonrig anonrig added the needs-benchmark-ci PRs that need a benchmark CI run. label Jun 21, 2024
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to file-system APIs and the fs module. needs-ci PRs that need a full CI run. labels Jun 21, 2024
@anonrig
anonrig force-pushed the improve-existssync branch from 21d2b35 to e2b3156 Compare June 21, 2024 20:08
joyeecheung
joyeecheung reviewed Jun 21, 2024
View reviewed changes

@joyeecheung joyeecheung left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you run the benchmarks under the benchmark folder? It’s not very reliable to benchmark a single fs call within a script using hyperfine since Node.js itself can take a lot more time to start up than doing that one fs call. (Perhaps around 40ms of that 48-47ms can be Node.js startup)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

tniessen reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread src/node_file.cc Resolved

Uh oh!

There was an error while loading. Please reload this page.

@anonrig

anonrig commented Jun 21, 2024

Copy link
Copy Markdown
Member Author

Have you run the benchmarks under the benchmark folder? It’s not very reliable to benchmark a single fs call within a script using hyperfine since Node.js itself can take a lot more time to start up than doing that one fs call. (Perhaps around 40ms of that 48-47ms can be Node.js startup)

I agree. Unfortunately, I couldn't figure out some weird npm issue on my Windows machine, and couldn't make a progress due to the slowness of the benchmarks (even if I decreased the n).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig
anonrig force-pushed the improve-existssync branch 2 times, most recently from 05866e2 to a7722d0 Compare June 21, 2024 20:24
@anonrig

anonrig commented Jun 21, 2024

Copy link
Copy Markdown
Member Author

I've opened an issue to keep track of the Windows issue I'm facing: #53538

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig anonrig added windows Issues and PRs related to the Windows platform. request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. labels Jun 21, 2024
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Jun 21, 2024
@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
jasnell
jasnell approved these changes Jun 21, 2024
View reviewed changes
tniessen
tniessen reviewed Jun 22, 2024
View reviewed changes

@tniessen tniessen left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @joyeecheung that this should be benchmarked properly before being merged. (Then again, benchmarking read-only fs operations is mostly pointless since the kernel will cache all the I/O and the results will not reflect real-world performance gains.)

Especially for symlinks, we are now not executing 2 different operating-system calls, and only one.

Did you test this hypothesis, or is it guaranteed by std::filesystem::exists?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread src/node_file.cc Resolved

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Jun 22, 2024

Copy link
Copy Markdown
Member

I agree. Unfortunately, I couldn't figure out some weird npm issue on my Windows machine, and couldn't make a progress due to the slowness of the benchmarks (even if I decreased the n).

Why would you need npm access to run the benchmarks? I think they can simply be run as

out/Release/node benchmark/compare.js --old ./node_main --new out/Release/node --filter existsSync fs > fs.csv

?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Jun 22, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Oh I see in #53538 you were trying to run node-benchmark-compare. You could just send that over to any machine that can run node-benchmark-compare to analyze the data, the csv is portable.

(Or you could run the R version with data.csv | Rscript benchmark/compare.R though I imagine getting Rscript to work on Windows can also be quite a task....or maybe not?)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@joyeecheung

joyeecheung commented Jun 22, 2024

Copy link
Copy Markdown
Member

Microbenchmark CI https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1571/

anonrig reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig

anonrig commented Jun 23, 2024

Copy link
Copy Markdown
Member Author

Microbenchmark CI https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1571/

It seems, this PR is 15% faster

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Jun 24, 2024

Copy link
Copy Markdown
Member

It seems, this PR is 15% faster

That only applies to the non-existing part. The question for me is what is more likely: existing or non-existing?

                                                          confidence improvement accuracy (*)   (**)  (***)
fs/bench-existsSync.js n=1000000 type='existing'                 ***     -0.92 %       ±0.29% ±0.39% ±0.51%
fs/bench-existsSync.js n=1000000 type='non-existing'             ***     15.26 %       ±1.16% ±1.55% ±2.04%
fs/bench-existsSync.js n=1000000 type='non-flat-existing'        ***      2.50 %       ±0.39% ±0.53% ±0.69%
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tniessen tniessen mentioned this pull request Jun 24, 2024
doc: clarify that fs.exists() may return false for existing symlink #53566
Merged
@anonrig
anonrig force-pushed the improve-existssync branch from a7722d0 to a4b05d2 Compare June 24, 2024 16:44
@anonrig

anonrig commented Jun 24, 2024

Copy link
Copy Markdown
Member Author

That only applies to the non-existing part. The question for me is what is more likely: existing or non-existing?

@BridgeAR for unix, the difference is 15% but on Windows, but on windows, it reduces the number of OS calls.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
@anonrig

anonrig commented Jun 24, 2024

Copy link
Copy Markdown
Member Author

Did you test this hypothesis, or is it guaranteed by std::filesystem::exists?

I was referencing the cppreference (https://en.cppreference.com/w/cpp/filesystem/exists)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tniessen

tniessen commented Jun 24, 2024

Copy link
Copy Markdown
Member

Did you test this hypothesis, or is it guaranteed by std::filesystem::exists?

I was referencing the cppreference (https://en.cppreference.com/w/cpp/filesystem/exists)

I don't see anything in that reference that would guarantee that the standard library function makes a single system call (or the Windows equivalent) and not multiple system calls.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
@anonrig
anonrig force-pushed the improve-existssync branch from a4b05d2 to 5c504b8 Compare June 26, 2024 19:14
@anonrig
fs: improve ExistsSync performance
0c05f7c
@anonrig
anonrig force-pushed the improve-existssync branch from 5c504b8 to 0c05f7c Compare June 26, 2024 19:15
@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
@nodejs-github-bot

nodejs-github-bot commented Jun 26, 2024

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/59985/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

nodejs-github-bot commented Jul 8, 2024

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60177/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig anonrig closed this Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jasnell jasnell jasnell approved these changes
@tniessen tniessen tniessen left review comments
@joyeecheung joyeecheung joyeecheung left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to file-system APIs and the fs module. needs-benchmark-ci PRs that need a benchmark CI run. needs-ci PRs that need a full CI run. windows Issues and PRs related to the Windows platform.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@anonrig @nodejs-github-bot @joyeecheung @BridgeAR @tniessen @jasnell
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.