GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — test_runner: fix test runner concurrency by MoLow · Pull Request #47675 · nodejs/node

Conversation

@MoLow

@MoLow MoLow commented Apr 22, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Fixes: #47365
Fixes: #47696

SafePromiseAllSettledReturnVoid runs sequentially before this fix :(
this makes me wonder if we should add a benchmark for the test runner

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
bricss and tonivj5 reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@MoLow
MoLow requested a review from aduh95 April 22, 2023 20:43
@nodejs-github-bot nodejs-github-bot added the needs-ci PRs that need a full CI run. label Apr 22, 2023
benjamingr
benjamingr approved these changes Apr 22, 2023
View reviewed changes
@MoLow

MoLow commented Apr 22, 2023

Copy link
Copy Markdown
Member Author

CC @nodejs/performance Who can help me with creating a benchmark? I never have before
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow MoLow added performance Issues and PRs related to the performance of Node.js. test_runner Issues and PRs related to the test runner subsystem. labels Apr 22, 2023
@tniessen

tniessen commented Apr 22, 2023

Copy link
Copy Markdown
Member

SafePromiseAllSettledReturnVoid runs sequentially before this fix

Is this the relevant invocation?

node/lib/internal/test_runner/runner.js

Lines 390 to 397 in 0c5f253

const runFiles = () => {
root.harness.bootstrapComplete = true;
return SafePromiseAllSettledReturnVoid(testFiles, (path) => {
const subtest = runTestFile(path, root, inspectPort, filesWatcher);
runningSubtests.set(path, subtest);
return subtest;
});
};

this makes me wonder if we should add a benchmark for the test runner

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.
MoLow reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow

MoLow commented Apr 22, 2023

Copy link
Copy Markdown
Member Author

Is this the relevant invocation?

yes

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.

I will add a test
tniessen reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@tniessen

tniessen commented Apr 22, 2023

Copy link
Copy Markdown
Member

@aduh95 In #45175, was the main motivation behind SafePromiseAllSettledReturnVoid not having to allocate an array for results, and so that the caller automatically obtains a Promise<void>? If so, this appears to be the only call site, so maybe we can just use SafePromiseAllSettled instead and return the Promise<void> from the test runner instead?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
tniessen
tniessen reviewed Apr 22, 2023
View reviewed changes

@tniessen tniessen left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the number of concurrent tests guaranteed to be at least two? I thought it depended on os.availableParallelism(), in which case it looks like this test might fail when only one or two CPU cores are assigned to the process group.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
MoLow reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@MoLow MoLow added the commit-queue-squash PRs the Commit Queue should land as one squashed commit. label Apr 22, 2023
@MoLow

MoLow commented Apr 22, 2023

Copy link
Copy Markdown
Member Author

this fix introduced a race condition in tests outputs being reported out of order, so I fixed that as well
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
MoLow
MoLow commented Apr 22, 2023
View reviewed changes
Comment thread test/parallel/test-runner-concurrency.js Outdated

@MoLow MoLow Apr 22, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cjihrig please confirm this skip makes sense

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@cjihrig cjihrig Apr 23, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense. I think you could verify by replacing the use of os.availableParallelism() in the test runner code with 1 or 2 and seeing if the test hangs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
mcollina
mcollina approved these changes Apr 23, 2023
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@MoLow MoLow added request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. author ready PRs with CI started, the required approvals, and no outstanding review comments. labels Apr 23, 2023
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Apr 23, 2023
@nodejs-github-bot

nodejs-github-bot commented Apr 23, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/51437/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot mentioned this pull request Apr 24, 2023
CI Reliability 2023-04-24 nodejs/reliability#545
Open
39 tasks
@MoLow MoLow mentioned this pull request Apr 24, 2023
test_runner: support combining coverage reports #47686
Merged
@MoLow
test_runner: fix test counting
8129d86
@MoLow
MoLow force-pushed the fix-test-runner-concurrency branch from e1dbbd2 to bbc6eaf Compare April 24, 2023 18:26
@MoLow

MoLow commented Apr 24, 2023

Copy link
Copy Markdown
Member Author

@cjihrig can you take a look? I've had to fix multiple race conditions in the test runner.
this also depends on #47699 to accurately count tests
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow

MoLow commented Apr 24, 2023

Copy link
Copy Markdown
Member Author

@benjamingr @mcollina PTAL
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow MoLow added commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. and removed commit-queue-squash PRs the Commit Queue should land as one squashed commit. labels Apr 24, 2023
cjihrig
cjihrig reviewed Apr 24, 2023
View reviewed changes
Comment thread lib/internal/test_runner/tests_stream.js Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread lib/internal/test_runner/runner.js Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread lib/internal/test_runner/runner.js Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread lib/internal/test_runner/runner.js Outdated

@cjihrig cjihrig Apr 24, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic in this function (and even the function signature) seems like it should be part of #handleReportItem(). Is it possible to handle this there?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@MoLow MoLow Apr 24, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handleReportItem happens only in order of reporting (using this.isClearToSend()), however, #skipReporting returns wrong values if it is called after the child process has completed but the test has not yet been reported

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@MoLow
test_runner: fix test runner concurrency
17e3338
@MoLow
MoLow force-pushed the fix-test-runner-concurrency branch from bbc6eaf to 17e3338 Compare April 24, 2023 19:25
@MoLow MoLow mentioned this pull request Apr 24, 2023
test_runner: fix test counting #47699
Closed
@nodejs-github-bot

nodejs-github-bot commented Apr 24, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/51450/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow

MoLow commented Apr 24, 2023

Copy link
Copy Markdown
Member Author

🟢 CI: https://ci.nodejs.org/job/node-test-pull-request/51451/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@MoLow MoLow added the commit-queue PRs queued for automated landing through the Commit Queue. label Apr 24, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Apr 24, 2023
@nodejs-github-bot

nodejs-github-bot commented Apr 24, 2023

Copy link
Copy Markdown
Collaborator

Landed in 8becacb...dccd25e
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@nodejs-github-bot nodejs-github-bot closed this Apr 24, 2023
nodejs-github-bot pushed a commit that referenced this pull request Apr 24, 2023
@MoLow @nodejs-github-bot
test_runner: fix test counting
dac0263 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
nodejs-github-bot pushed a commit that referenced this pull request Apr 24, 2023
@MoLow @nodejs-github-bot
test_runner: fix test runner concurrency
dccd25e 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@MoLow
MoLow deleted the fix-test-runner-concurrency branch April 24, 2023 22:33
This was referenced Apr 25, 2023
CI Reliability 2023-04-25 nodejs/reliability#546
Open
CI Reliability 2023-04-26 nodejs/reliability#547
Open
@tniessen tniessen mentioned this pull request Apr 26, 2023
concurrent tests are slow #47365
Closed
This was referenced Apr 27, 2023
CI Reliability 2023-04-27 nodejs/reliability#548
Open
CI Reliability 2023-04-28 nodejs/reliability#549
Open
@github-actions github-actions Bot mentioned this pull request Apr 29, 2023
CI Reliability 2023-04-29 nodejs/reliability#550
Open
32 tasks
@github-actions github-actions Bot mentioned this pull request Apr 30, 2023
CI Reliability 2023-04-30 nodejs/reliability#551
Open
33 tasks
targos pushed a commit that referenced this pull request May 2, 2023
@MoLow @targos
test_runner: fix test counting
2d7cac0 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
targos pushed a commit that referenced this pull request May 2, 2023
@MoLow @targos
test_runner: fix test runner concurrency
40b3879 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@targos targos mentioned this pull request May 2, 2023
v20.1.0 release proposal #47820
Merged
danielleadams pushed a commit that referenced this pull request Jul 6, 2023
@MoLow @danielleadams
test_runner: fix test counting
003b2d4 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
danielleadams pushed a commit that referenced this pull request Jul 6, 2023
@MoLow @danielleadams
test_runner: fix test runner concurrency
85edbc3 
PR-URL: #47675
Fixes: #47365
Fixes: #47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
MoLow added a commit to MoLow/node that referenced this pull request Jul 6, 2023
@MoLow
test_runner: fix test counting
3bbb1fc 
PR-URL: nodejs#47675
Fixes: nodejs#47365
Fixes: nodejs#47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
MoLow added a commit to MoLow/node that referenced this pull request Jul 6, 2023
@MoLow
test_runner: fix test runner concurrency
323881f 
PR-URL: nodejs#47675
Fixes: nodejs#47365
Fixes: nodejs#47696
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@danielleadams danielleadams mentioned this pull request Jul 10, 2023
v18.17.0 release proposal #48694
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tniessen tniessen tniessen left review comments
@mcollina mcollina mcollina approved these changes
@benjamingr benjamingr benjamingr approved these changes
@cjihrig cjihrig cjihrig approved these changes
@aduh95 aduh95 Awaiting requested review from aduh95

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. needs-ci PRs that need a full CI run. performance Issues and PRs related to the performance of Node.js. test_runner Issues and PRs related to the test runner subsystem.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

nested test counting is wrong concurrent tests are slow

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@MoLow @tniessen @nodejs-github-bot @mcollina @benjamingr @cjihrig
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.