GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — benchmark: rewrite the startup performance benchmark by joyeecheung · Pull Request #45746 · nodejs/node

Conversation

@joyeecheung

@joyeecheung joyeecheung commented Dec 5, 2022

Copy link
Copy Markdown
Member

benchmark: make benchmarks runnable in older versions of Node.js

Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

benchmark: rewrite the startup performance benchmark

The previous benchmark was inaccurate in several ways:

  • It previously measured "how many workers/child processes can finish
    loading in parallel in 1 second" which is prone to wrong
    interpretations. It's also not typical for users to spin up >20
    workers/child processes in parallel, which was what the benchmark
    usually ended up doing. The parallelism can also be greatly affected
    by system configurations.
  • The time spent on loading a Node.js binary for the first time varies
    greatly depending on system configurations, but it was inevitable
    for the benchmark to load Node.js multiple times to reduce the
    influence of fluctuations.

This patch rewrites the benchmark so that:

  • It now measures "how long it takes to finish 30 workers/child processes
    in serial" which is generally more in line with how other benchmarks
    are written and with the figures one gets from doing something like
    time node index.js or hyperfine 'node index.js'
  • It now warms up the loading of the Node.js instance to reduce the
    statistical outliers, so that we get more meaningful figures when
    trying to compare the startup performance of different Node.js
    binaries.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
joyeecheung added 2 commits December 5, 2022 11:17
@joyeecheung
benchmark: make benchmarks runnable in older versions of Node.js
996719f 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
@joyeecheung
benchmark: rewrite the startup performance benchmark
bc05a5f 
The previous benchmark was inaccurate in several ways:

- It previously measured "how many workers/child processes can finish
  loading in parallel in 1 second" which is prone to wrong
  interpretations. It's also not typical for users to spin up >20
  workers/child processes in parallel, which was what the benchmark
  usually ended up doing. The parallelism can also be greatly affected
  by system configurations.
- The time spent on loading a Node.js binary for the first time varies
  greatly depending on system configurations, but it was inevitable
  for the benchmark to load Node.js multiple times to reduce the
  influence of fluctuations.

This patch rewrites the benchmark so that:

- It now measures "how long it takes to finish 30 workers/child processes
  in serial" which is generally more in line with how other benchmarks
  are written and with the figures one gets from doing something like
  `time node index.js` or `hyperfine 'node index.js'`
- It now warms up the loading of the Node.js instance to reduce the
  statistical outliers, so that we get more meaningful figures when
  trying to compare the startup performance of different Node.js
  binaries.
@nodejs-github-bot

nodejs-github-bot commented Dec 5, 2022

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/startup
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@nodejs-github-bot nodejs-github-bot added the benchmark Issues and PRs related to Node.js benchmarks and benchmarking infrastructure. label Dec 5, 2022
@joyeecheung

joyeecheung commented Dec 5, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

New benchmark results from #45659, which is more in line with the results I see from hyperfine:

                                                                                     confidence improvement accuracy (*)   (**)  (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins'                -0.25 %       ±2.05% ±2.78% ±3.71%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon'                    ***     14.04 %       ±2.80% ±3.79% ±5.06%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins'                  0.19 %       ±2.10% ±2.84% ±3.79%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon'                     ***     16.86 %       ±2.67% ±3.61% ±4.82%

I'll recompile and see the new results from #45716
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@joyeecheung joyeecheung mentioned this pull request Dec 5, 2022
build: disable v8 snapshot compression #45716
Closed
@joyeecheung

joyeecheung commented Dec 5, 2022

Copy link
Copy Markdown
Member Author

New benchmark numbers from #45716


                                                                                     confidence improvement accuracy (*)   (**)  (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins'        ***     12.28 %       ±1.13% ±1.52% ±2.00%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon'                    ***     17.39 %       ±1.12% ±1.50% ±1.97%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins'         ***     14.54 %       ±0.77% ±1.03% ±1.35%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon'                     ***     18.02 %       ±1.02% ±1.36% ±1.79%
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@joyeecheung joyeecheung mentioned this pull request Dec 5, 2022
bootstrap: lazy load non-essential modules #45659
Merged
legendecas
legendecas reviewed Dec 5, 2022
View reviewed changes
Comment thread benchmark/misc/startup.js Resolved

Uh oh!

There was an error while loading. Please reload this page.
joyeecheung added 2 commits December 5, 2022 22:59
@joyeecheung
fixup! benchmark: rewrite the startup performance benchmark
4b1b762
@joyeecheung
fixup! fixup! benchmark: rewrite the startup performance benchmark
9f82938
legendecas
legendecas approved these changes Dec 6, 2022
View reviewed changes
@legendecas

legendecas commented Dec 6, 2022

Copy link
Copy Markdown
Member

Linter is still complaining.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@joyeecheung
fixup! fixup! fixup! benchmark: rewrite the startup performance bench…
ee2cbe3 
…mark
@nodejs-github-bot

nodejs-github-bot commented Dec 6, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48305/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@nodejs-github-bot

nodejs-github-bot commented Dec 6, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48313/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot mentioned this pull request Dec 8, 2022
CI Reliability 2022-12-08 nodejs/reliability#451
Open
8 tasks
@nodejs-github-bot

nodejs-github-bot commented Dec 8, 2022

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48341/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@joyeecheung joyeecheung added the review wanted PRs that need review. label Dec 8, 2022
@joyeecheung

joyeecheung commented Dec 8, 2022

Copy link
Copy Markdown
Member Author

@nodejs/performance Can I have some reviews please?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
anonrig
anonrig approved these changes Dec 8, 2022
View reviewed changes
@anonrig anonrig added performance Issues and PRs related to the performance of Node.js. author ready PRs with CI started, the required approvals, and no outstanding review comments. commit-queue-squash PRs the Commit Queue should land as one squashed commit. commit-queue PRs queued for automated landing through the Commit Queue. labels Dec 8, 2022
addaleax
addaleax approved these changes Dec 8, 2022
View reviewed changes
@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Dec 8, 2022
@nodejs-github-bot
nodejs-github-bot merged commit 2b9d3b2 into nodejs:main Dec 8, 2022
@nodejs-github-bot

nodejs-github-bot commented Dec 8, 2022

Copy link
Copy Markdown
Collaborator

Landed in 2b9d3b2
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot mentioned this pull request Dec 9, 2022
CI Reliability 2022-12-09 nodejs/reliability#452
Open
5 tasks
This was referenced Dec 10, 2022
CI Reliability 2022-12-10 nodejs/reliability#453
Open
CI Reliability 2022-12-11 nodejs/reliability#454
Open
targos pushed a commit that referenced this pull request Dec 12, 2022
@joyeecheung @targos
benchmark: make benchmarks runnable in older versions of Node.js
6f0b1a1 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
@targos targos mentioned this pull request Dec 12, 2022
v19.3.0 release proposal #45831
Merged
targos pushed a commit that referenced this pull request Dec 13, 2022
@joyeecheung @targos
benchmark: make benchmarks runnable in older versions of Node.js
102c2dc 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
@danielleadams danielleadams mentioned this pull request Dec 30, 2022
v18.13.0 release proposal #46025
Merged
danielleadams pushed a commit that referenced this pull request Dec 30, 2022
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
199b4f1 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
danielleadams pushed a commit that referenced this pull request Jan 3, 2023
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
5e167bd 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
danielleadams pushed a commit that referenced this pull request Jan 5, 2023
@joyeecheung @danielleadams
benchmark: make benchmarks runnable in older versions of Node.js
1442010 
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

PR-URL: #45746
Reviewed-By: Chengzhong Wu <[email protected]>
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@addaleax addaleax addaleax approved these changes
@anonrig anonrig anonrig approved these changes
@legendecas legendecas legendecas approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. benchmark Issues and PRs related to Node.js benchmarks and benchmarking infrastructure. commit-queue-squash PRs the Commit Queue should land as one squashed commit. performance Issues and PRs related to the performance of Node.js. review wanted PRs that need review.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@joyeecheung @nodejs-github-bot @legendecas @addaleax @anonrig
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.