GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fs: improve `readFileSync` with file descriptors by anonrig · Pull Request #49691 · nodejs/node

Conversation

@anonrig

@anonrig anonrig commented Sep 17, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This particular change applies to: fs.readFileSync(fs.openSync(__filename), 'utf8')

My local benchmarks are as follows:

fs/readFileSync.js n=10000 hasFileDescriptor='false' path='existing' encoding='undefined'                     0.73 %       ±1.65% ±2.20% ±2.86%
fs/readFileSync.js n=10000 hasFileDescriptor='false' path='existing' encoding='utf8'                          0.05 %       ±1.74% ±2.32% ±3.02%
fs/readFileSync.js n=10000 hasFileDescriptor='false' path='non-existing' encoding='undefined'        ***     62.27 %       ±2.82% ±3.77% ±4.95%
fs/readFileSync.js n=10000 hasFileDescriptor='false' path='non-existing' encoding='utf8'             ***     60.93 %       ±2.72% ±3.64% ±4.78%
fs/readFileSync.js n=10000 hasFileDescriptor='true' path='existing' encoding='undefined'                      0.22 %       ±1.47% ±1.96% ±2.56%
fs/readFileSync.js n=10000 hasFileDescriptor='true' path='existing' encoding='utf8'                  ***     88.18 %       ±2.56% ±3.43% ±4.49%
fs/readFileSync.js n=10000 hasFileDescriptor='true' path='non-existing' encoding='undefined'                 -3.31 %       ±4.32% ±5.80% ±7.65%
fs/readFileSync.js n=10000 hasFileDescriptor='true' path='non-existing' encoding='utf8'                      -3.18 %       ±3.58% ±4.79% ±6.30%

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1413

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vcapretz, bhavitsharma, bricss, H4ad, Atanur, joaopcm, arsaccol, atlowChemi, daniellockyer, tundeLayan, and 13 more reacted with heart emoji
All reactions
  • ❤️ 23 reactions
@anonrig anonrig added the performance Issues and PRs related to the performance of Node.js. label Sep 17, 2023
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to file-system APIs and the fs module. needs-ci PRs that need a full CI run. labels Sep 17, 2023
@anonrig anonrig mentioned this pull request Sep 17, 2023
Performance of node:fs nodejs/performance#106
Open
@rluvaton rluvaton added the needs-benchmark-ci PRs that need a benchmark CI run. label Sep 17, 2023
@rluvaton

rluvaton commented Sep 18, 2023

Copy link
Copy Markdown
Member

Benchmark CI:
https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1397/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bnoordhuis

bnoordhuis commented Sep 18, 2023

Copy link
Copy Markdown
Member

Test failures look related.

Correctness issue: file descriptors are int32, not uint32 (also on Windows, because they go through libuv.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig

anonrig commented Sep 18, 2023

Copy link
Copy Markdown
Member Author

@bnoordhuis We validate file descriptors on lib/fs.js by checking if it is Uint32 (ref: https://github.com/nodejs/node/blob/main/lib/fs.js#L204).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bnoordhuis

bnoordhuis commented Sep 18, 2023

Copy link
Copy Markdown
Member

Apparently that goes back to commit 0803962 from 2015... still wrong though. :-)

(Mostly harmless although technically signed integer overflow is UB in C++.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig

anonrig commented Sep 18, 2023

Copy link
Copy Markdown
Member Author

still wrong though. :-)

Nice find! I'll update the pull request.

@RafaelGSS I'm getting some weird failed test cases where stdout and stderr are both empty in the failing cases, but returning a status of 1. Any idea?

AssertionError [ERR_ASSERTION]
    at Object.<anonymous> (/Users/runner/work/node/node/test/parallel/test-permission-fs-read.js:42:10)
    at Module._compile (node:internal/modules/cjs/loader:1264:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1318:10)
    at Module.load (node:internal/modules/cjs/loader:1113:32)
    at Module._load (node:internal/modules/cjs/loader:954:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:80:12)
    at node:internal/main/run_main_module:23:47 {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: null,
  expected: 0,
  operator: 'strictEqual'
}
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig
anonrig force-pushed the fs-read-file-sync-fd branch 2 times, most recently from 0dfcfa4 to de56591 Compare September 18, 2023 19:53
@anonrig
anonrig requested a review from RafaelGSS September 20, 2023 15:06
RafaelGSS
RafaelGSS previously requested changes Sep 21, 2023
View reviewed changes
Comment thread src/node_file.cc Resolved

Uh oh!

There was an error while loading. Please reload this page.

@anonrig
anonrig requested a review from RafaelGSS September 21, 2023 19:29
@anonrig anonrig mentioned this pull request Sep 21, 2023
fs: fix file descriptor validator #49752
Merged
anonrig added 2 commits September 24, 2023 16:49
@anonrig
fs: improve readFileSync with file descriptors
28cf7ff
@anonrig
fixup! fs: improve readFileSync with file descriptors
a415654
@anonrig
anonrig force-pushed the fs-read-file-sync-fd branch from 6fea27f to a415654 Compare September 24, 2023 20:49
@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Sep 24, 2023
@anonrig

anonrig commented Sep 24, 2023 •
edited by targos
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1413

09:17:21                                                                                               confidence improvement accuracy (*)    (**)   (***)
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='false' path='existing' encoding='undefined'                    -2.80 %       ±8.78% ±11.68% ±15.21%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='false' path='existing' encoding='utf8'                         -6.68 %       ±9.49% ±12.63% ±16.44%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='false' path='non-existing' encoding='undefined'                -3.47 %       ±9.74% ±12.97% ±16.88%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='false' path='non-existing' encoding='utf8'                     -5.20 %       ±8.47% ±11.26% ±14.66%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='true' path='existing' encoding='undefined'                     -1.99 %       ±4.82%  ±6.41%  ±8.34%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='true' path='existing' encoding='utf8'                  ***    126.70 %      ±16.25% ±21.75% ±28.56%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='true' path='non-existing' encoding='undefined'                 -1.31 %       ±6.23%  ±8.29% ±10.79%
09:17:21 fs/readFileSync.js n=10000 hasFileDescriptor='true' path='non-existing' encoding='utf8'              ***    143.95 %      ±18.72% ±25.19% ±33.36%
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Sep 24, 2023
@nodejs-github-bot

nodejs-github-bot commented Sep 24, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/54200/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@anonrig

anonrig commented Sep 24, 2023

Copy link
Copy Markdown
Member Author

@nodejs/cpp-reviewers @nodejs/fs Can you review?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bnoordhuis
bnoordhuis reviewed Sep 25, 2023
View reviewed changes
Comment thread src/node_file.cc
CHECK_EQ(0, uv_fs_close(nullptr, &req, file, nullptr));
FS_SYNC_TRACE_END(close);
}
uv_fs_req_cleanup(&req);

@bnoordhuis bnoordhuis Sep 25, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For correctness reasons this ought to be done after the uv_fs_read() call on line 2590. It works okay now because libuv doesn't allocate for a single buffer (i.e. this cleanup call is a no-op) but that's a lucky accident.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@anonrig anonrig Sep 25, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that mean that on every while loop iteration (for uv_fs_read), we need to call uv_fs_req_cleanup?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@bnoordhuis bnoordhuis Sep 26, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@RafaelGSS
RafaelGSS dismissed their stale review September 25, 2023 17:03

resolved

Qard
Qard approved these changes Sep 25, 2023
View reviewed changes
@anonrig anonrig added the commit-queue PRs queued for automated landing through the Commit Queue. label Sep 25, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Sep 25, 2023
@nodejs-github-bot

nodejs-github-bot commented Sep 25, 2023

Copy link
Copy Markdown
Collaborator

Landed in f16f41c

styfle reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@targos

targos commented Sep 26, 2023

Copy link
Copy Markdown
Member

@anonrig Please always post the benchmark CI results in the PR (for example by editing the comment with the link like I just did). Jenkins results are not permanent.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ruyadorno pushed a commit that referenced this pull request Sep 28, 2023
@anonrig @ruyadorno
fs: improve readFileSync with file descriptors
07d0518
PR-URL: #49691
Reviewed-By: Stephen Belanger <[email protected]>
@ruyadorno ruyadorno mentioned this pull request Sep 28, 2023
v20.7.1 proposal #49917
Closed
ruyadorno pushed a commit that referenced this pull request Sep 28, 2023
@anonrig @ruyadorno
fs: improve readFileSync with file descriptors
31702c9
PR-URL: #49691
Reviewed-By: Stephen Belanger <[email protected]>
@ruyadorno ruyadorno mentioned this pull request Sep 28, 2023
v20.8.0 proposal #49932
Merged
This was referenced Oct 2, 2023
binding: mock readFileSync tschaub/mock-fs#378
Closed
Fix tests on Node 20.8.0+ getgauge/gauge-js#603
Merged
@richardlau richardlau added the dont-land-on-v18.x label Mar 15, 2024
This was referenced Mar 15, 2024
fs: fix edge case in readFileSync utf8 fast path #52101
Merged
fs.writeFileSync in Node 20 failed to write file in TMPDIR #52079
Closed
nodejs-github-bot pushed a commit that referenced this pull request Mar 18, 2024
@richardlau
fs: fix edge case in readFileSync utf8 fast path
39f1b89
Fix a file permissions regression when `fs.readFileSync()` is called in
append mode on a file that does not already exist introduced by the
fast path for utf8 encoding.

PR-URL: #52101
Fixes: #52079
Refs: #49691
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request May 2, 2024
@richardlau @marco-ippolito
fs: fix edge case in readFileSync utf8 fast path
0c82740
Fix a file permissions regression when `fs.readFileSync()` is called in
append mode on a file that does not already exist introduced by the
fast path for utf8 encoding.

PR-URL: #52101
Fixes: #52079
Refs: #49691
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request May 3, 2024
@richardlau @marco-ippolito
fs: fix edge case in readFileSync utf8 fast path
0a9910c
Fix a file permissions regression when `fs.readFileSync()` is called in
append mode on a file that does not already exist introduced by the
fast path for utf8 encoding.

PR-URL: #52101
Fixes: #52079
Refs: #49691
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
@tniessen tniessen mentioned this pull request Apr 9, 2025
Memory leak in node::fs::ReadFileUtf8(const FunctionCallbackInfo<Value>& args) when args[0] is a string. #57800
Closed
@Justin-Nietzel Justin-Nietzel mentioned this pull request Apr 9, 2025
Free req.file.pathw in fs::ReadFileUtf8(). #57811
Closed
Justin-Nietzel added a commit to Justin-Nietzel/node-issue-57800 that referenced this pull request Apr 10, 2025
@Justin-Nietzel
fs: add missing call to uv_fs_req_cleanup
a00fc51
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR nodejs#49691.

Fixes: nodejs#57800
jasnell pushed a commit that referenced this pull request Apr 12, 2025
@Justin-Nietzel @jasnell
fs: add missing call to uv_fs_req_cleanup
6a7b005
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
This was referenced Apr 19, 2025
For correctness reasons this ought to be done after the uv_fs_read() call on line 2590. It works okay now because libuv doesn't allocate for a single buffer (i.e. this cleanup call is a no-op) but that's a lucky accident. actions/runner#3815
Closed
For correctness reasons, this ought to be done after the uv_fs_read() call on line 2590. It works okay now because libuv doesn't allocate for a single buffer (i.e., this cleanup call is a no-op) but that's a lucky accident. actions/runner#3816
Closed
RafaelGSS pushed a commit that referenced this pull request May 1, 2025
@Justin-Nietzel @RafaelGSS
fs: add missing call to uv_fs_req_cleanup
ea95ec6
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
RafaelGSS pushed a commit that referenced this pull request May 2, 2025
@Justin-Nietzel @RafaelGSS
fs: add missing call to uv_fs_req_cleanup
52e4967
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
aduh95 pushed a commit that referenced this pull request May 6, 2025
@Justin-Nietzel @aduh95
fs: add missing call to uv_fs_req_cleanup
ea4475a
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
aduh95 pushed a commit that referenced this pull request May 6, 2025
@Justin-Nietzel @aduh95
fs: add missing call to uv_fs_req_cleanup
e24a3e2
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
RafaelGSS pushed a commit that referenced this pull request May 14, 2025
@Justin-Nietzel @RafaelGSS
fs: add missing call to uv_fs_req_cleanup
9e13bf0
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
CVE-ID: CVE-2025-23165
RafaelGSS pushed a commit that referenced this pull request May 14, 2025
@Justin-Nietzel @RafaelGSS
fs: add missing call to uv_fs_req_cleanup
9f403e9
Always call uv_fs_req_cleanup after calling uv_fs_open instead of just
when uv_fs_open returns a negative result. I referenced ReadFileSync
from node:js2c when making this change.

https://github.com/bnoordhuis made the same suggestion based on the
PR #49691.

Fixes: #57800
PR-URL: #57811
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
CVE-ID: CVE-2025-23165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bnoordhuis bnoordhuis bnoordhuis left review comments
@Qard Qard Qard approved these changes
@RafaelGSS RafaelGSS Awaiting requested review from RafaelGSS

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to file-system APIs and the fs module. needs-benchmark-ci PRs that need a benchmark CI run. needs-ci PRs that need a full CI run. performance Issues and PRs related to the performance of Node.js.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@anonrig @rluvaton @bnoordhuis @nodejs-github-bot @targos @Qard @RafaelGSS @richardlau
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.