GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fs: improve promise based readFile performance for big files by BridgeAR · Pull Request #44295 · nodejs/node

Conversation

@BridgeAR

@BridgeAR BridgeAR commented Aug 19, 2022

Copy link
Copy Markdown
Member

This significantly reduces the peak memory for the promise
based readFile operation by reusing a single memory chunk after
each read and strinigifying that chunk immediately.

Refs: https://github.com/nodejs/node/discussions/44239#discussioncomment-3428693

Signed-off-by: Ruben Bridgewater [email protected]

Benchmark with a few runs:

                                                              confidence improvement accuracy (*)    (**)   (***)
fs/readfile-promises.js concurrent=1 len=1024 duration=5                     -1.18 %       ±7.07%  ±9.67% ±13.12%
fs/readfile-promises.js concurrent=1 len=16777216 duration=5                 -2.11 %       ±3.34%  ±4.56%  ±6.20%
fs/readfile-promises.js concurrent=1 len=33554432 duration=5         ***     59.48 %       ±5.41%  ±7.45% ±10.25%
fs/readfile-promises.js concurrent=1 len=4194304 duration=5          ***      9.95 %       ±5.10%  ±6.93%  ±9.32%
fs/readfile-promises.js concurrent=1 len=524288 duration=5                    1.88 %       ±6.77%  ±9.22% ±12.45%
fs/readfile-promises.js concurrent=1 len=8388608 duration=5           **      6.18 %       ±4.19%  ±5.70%  ±7.69%
fs/readfile-promises.js concurrent=10 len=1024 duration=5                    -1.21 %       ±8.11% ±11.12% ±15.15%
fs/readfile-promises.js concurrent=10 len=16777216 duration=5        ***      9.72 %       ±5.07%  ±6.92%  ±9.36%
fs/readfile-promises.js concurrent=10 len=33554432 duration=5        ***     26.58 %       ±6.29%  ±8.56% ±11.56%
fs/readfile-promises.js concurrent=10 len=4194304 duration=5          **      9.66 %       ±5.88%  ±8.00% ±10.77%
fs/readfile-promises.js concurrent=10 len=524288 duration=5                   1.38 %       ±5.49%  ±7.47% ±10.07%
fs/readfile-promises.js concurrent=10 len=8388608 duration=5         ***     16.84 %       ±4.35%  ±5.93%  ±8.02%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 12 comparisons, you can thus
expect the following amount of false-positive results:
  0.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.12 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bricss and tonivj5 reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@BridgeAR
BridgeAR requested a review from mcollina August 19, 2022 15:12
@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to file-system APIs and the fs module. needs-ci PRs that need a full CI run. labels Aug 19, 2022
mcollina
mcollina approved these changes Aug 19, 2022
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@BridgeAR
BridgeAR marked this pull request as draft August 19, 2022 18:50
@BridgeAR
BridgeAR force-pushed the improve-fs-promises-readfile branch from 744bbd9 to e72802b Compare August 20, 2022 01:10
@BridgeAR
BridgeAR marked this pull request as ready for review August 20, 2022 01:11
@BridgeAR

BridgeAR commented Aug 20, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

We might consider a similar fix for #41435. If I checked correct, it outperforms the regular readFile variant for big files.
The lower peak memory and chunked stringify operations is better than less C++/JS boundary crossings.

mcollina and bricss reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Aug 20, 2022

Copy link
Copy Markdown
Member

Does this also impact "old" callback- based readFile?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Aug 20, 2022

Copy link
Copy Markdown
Member Author

Does this also impact "old" callback- based readFile?

No. This is solely about the promise based readFile with encodings. The non encoding part is also not impacted.
The implementation of the callback-based readFile differs from the promise based one.

23:39:15                                                                               confidence improvement accuracy (*)    (**)   (***)
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=1024 duration=5              *     -9.10 %       ±8.50% ±11.31% ±14.73%
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=16777216 duration=5         **      7.96 %       ±5.27%  ±7.02%  ±9.16%
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=33554432 duration=5        ***     63.48 %       ±6.59%  ±8.81% ±11.56%
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=4194304 duration=5                  2.20 %       ±3.73%  ±4.97%  ±6.47%
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=524288 duration=5                  -0.89 %       ±2.44%  ±3.25%  ±4.23%
23:39:15 fs/readfile-promises.js concurrent=10 encoding='utf8' len=8388608 duration=5                  4.81 %       ±5.08%  ±6.77%  ±8.84%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=1024 duration=5                     -0.63 %       ±2.87%  ±3.82%  ±4.98%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=16777216 duration=5         ***     -9.93 %       ±4.71%  ±6.28%  ±8.21%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=33554432 duration=5         ***     75.47 %      ±10.06% ±13.56% ±17.98%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=4194304 duration=5          ***    -11.12 %       ±5.45%  ±7.30%  ±9.61%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=524288 duration=5                   -2.80 %       ±5.00%  ±6.65%  ±8.67%
23:39:15 fs/readfile-promises.js concurrent=1 encoding='utf8' len=8388608 duration=5          ***      9.33 %       ±3.50%  ±4.66%  ±6.07%

I ran the benchmark on our machines and there seems to be a weird effect for some file sizes where there's a performance drop. I guess it has to do with how V8 handles some things internally.
Reads up to 512kb are not impacted (the one star is misleading) and big reads (all tests above 32mb) are always significantly faster. In-between there are some file sizes that profit and some where there's a ~10% performance hit.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Aug 20, 2022

Copy link
Copy Markdown
Member Author

Benchmark: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1176/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

LiviaMedeiros
LiviaMedeiros reviewed Aug 20, 2022
View reviewed changes
Comment thread lib/internal/fs/promises.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

mcollina
mcollina approved these changes Aug 22, 2022
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
jasnell
jasnell approved these changes Aug 28, 2022
View reviewed changes
@mcollina

mcollina commented Aug 29, 2022

Copy link
Copy Markdown
Member

@BridgeAR are you planning to do the same for the sync and callback versions too?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR
BridgeAR force-pushed the improve-fs-promises-readfile branch 3 times, most recently from 3f028d6 to 83562ca Compare October 4, 2022 23:06
@BridgeAR
fs: improve promise based readFile performance for big files
6c92996
This significantly reduces the peak memory for the promise
based readFile operation by reusing a single memory chunk after
each read and strinigifying that chunk immediately.

Signed-off-by: Ruben Bridgewater <[email protected]>
@BridgeAR
BridgeAR force-pushed the improve-fs-promises-readfile branch from 4d4997b to 6c92996 Compare October 4, 2022 23:07
@BridgeAR BridgeAR added request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. and removed needs-ci PRs that need a full CI run. labels Oct 5, 2022
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 5, 2022
@nodejs-github-bot

nodejs-github-bot commented Oct 5, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/47083/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

nodejs-github-bot commented Oct 5, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/47093/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Oct 5, 2022

Copy link
Copy Markdown
Member Author

are you planning to do the same for the sync and callback versions too?

@mcollina I suggest to merge this and I am looking into the other part afterwards.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot mentioned this pull request Oct 6, 2022
CI Reliability 2022-10-06 nodejs/reliability#396
Open
25 tasks
@nodejs-github-bot

nodejs-github-bot commented Oct 6, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/47101/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina mcollina added the commit-queue PRs queued for automated landing through the Commit Queue. label Oct 6, 2022
@mcollina

mcollina commented Oct 6, 2022

Copy link
Copy Markdown
Member

Yes, let's do it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Oct 6, 2022
@nodejs-github-bot

nodejs-github-bot commented Oct 6, 2022

Copy link
Copy Markdown
Collaborator
Commit Queue failed
- Loading data for nodejs/node/pull/44295
✔  Done loading data for nodejs/node/pull/44295
----------------------------------- PR info ------------------------------------
Title      fs: improve promise based readFile performance for big files (#44295)
Author     Ruben Bridgewater  (@BridgeAR)
Branch     BridgeAR:improve-fs-promises-readfile -> nodejs:main
Labels     fs
Commits    1
 - fs: improve promise based readFile performance for big files
Committers 1
 - Ruben Bridgewater 
PR-URL: https://github.com/nodejs/node/pull/44295
Reviewed-By: Matteo Collina 
Reviewed-By: James M Snell 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/44295
Reviewed-By: Matteo Collina 
Reviewed-By: James M Snell 
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last review:
   ⚠  - fs: improve promise based readFile performance for big files
   ℹ  This PR was created on Fri, 19 Aug 2022 15:12:52 GMT
   ✔  Approvals: 2
   ✔  - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/44295#pullrequestreview-1080779016
   ✔  - James M Snell (@jasnell) (TSC): https://github.com/nodejs/node/pull/44295#pullrequestreview-1087869731
   ✔  Last GitHub CI successful
   ℹ  Last Benchmark CI on 2022-08-20T10:00:08Z: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1176/
   ℹ  Last Full PR CI on 2022-10-06T13:18:58Z: https://ci.nodejs.org/job/node-test-pull-request/47101/
- Querying data for job/node-test-pull-request/47101/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu
https://github.com/nodejs/node/actions/runs/3198926242
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot nodejs-github-bot added the commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. label Oct 6, 2022
mcollina
mcollina approved these changes Oct 6, 2022
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mcollina mcollina added commit-queue PRs queued for automated landing through the Commit Queue. and removed commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. labels Oct 6, 2022
@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Oct 6, 2022
@nodejs-github-bot
nodejs-github-bot merged commit d3dd49f into nodejs:main Oct 6, 2022
@nodejs-github-bot

nodejs-github-bot commented Oct 6, 2022

Copy link
Copy Markdown
Collaborator

Landed in d3dd49f

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Oct 7, 2022
CI Reliability 2022-10-07 nodejs/reliability#397
Open
CI Reliability 2022-10-09 nodejs/reliability#399
Open
CI Reliability 2022-10-10 nodejs/reliability#400
Open
CI Reliability 2022-10-11 nodejs/reliability#401
Open
danielleadams pushed a commit that referenced this pull request Oct 11, 2022
@BridgeAR @danielleadams
fs: improve promise based readFile performance for big files
67fb765
This significantly reduces the peak memory for the promise
based readFile operation by reusing a single memory chunk after
each read and strinigifying that chunk immediately.

Signed-off-by: Ruben Bridgewater <[email protected]>
PR-URL: #44295
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: James M Snell <[email protected]>
@danielleadams danielleadams mentioned this pull request Oct 11, 2022
v18.11.0 proposal #44968
Merged
@BridgeAR BridgeAR mentioned this pull request Mar 13, 2024
readFile in promises very slow nodejs/performance#151
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LiviaMedeiros LiviaMedeiros LiviaMedeiros left review comments
@mcollina mcollina mcollina approved these changes
@jasnell jasnell jasnell approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

fs Issues and PRs related to file-system APIs and the fs module.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@BridgeAR @mcollina @nodejs-github-bot @jasnell @LiviaMedeiros
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.