GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — stream: complete pipeline with stdio by ronag · Pull Request #32373 · nodejs/node

Conversation

@ronag

@ronag ronag commented Mar 19, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

stdio (stderr & stdout) should for compatibility
reasons not be closed/end():ed. However, this
causes pipeline with a stdio destination to
never finish. This commit fixes this issue at
a performance cost.

Refs: #7606

Fixes: #32363

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bricss reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@ronag
stream: complete pipeline with stdio
571e867
stdio (stderr & stdout) should for compatibility
reasons not be closed/end():ed. However, this
causes pipeline with a stdio destination to
never finish. This commit fixes this issue at
a performance cost.

Refs: nodejs#7606

Fixes: nodejs#32363
@ronag
ronag requested a review from mcollina March 19, 2020 21:41
@ronag ronag added the stream Issues and PRs related to Node.js streams. label Mar 19, 2020
@ronag

ronag commented Mar 19, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

I'm not sure I understand/agree with not calling end() on stdio streams. However, this PR resolves the issue of pipeline not completing at a performance cost. This is the best solution I've found so far...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jasnell
jasnell approved these changes Mar 19, 2020
View reviewed changes
ronag added 2 commits March 19, 2020 22:46
@ronag
fixup: isStdio helper
ab76605
@ronag
fixup: test
84cf2aa
@ronag

ronag commented Mar 19, 2020

Copy link
Copy Markdown
Member Author

@vweevers

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ronag added 2 commits March 19, 2020 22:47
@ronag
fixup: typo
b2925d8
@ronag
fixup: test
bb882a2
BridgeAR
BridgeAR reviewed Mar 19, 2020
View reviewed changes
Comment thread lib/internal/streams/pipeline.js Outdated
err ? reject(err) : resolve();
});
})
]);

@BridgeAR BridgeAR Mar 19, 2020

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is likely faster to use the callbacks and manually handle everything. It could be wrapped in a single promise to allow to await the result for simplicity. Alternatively just also handle the finish call separately.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ronag ronag Mar 19, 2020

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is likely faster to use the callbacks and manually handle everything. It could be wrapped in a single promise to allow to await the result for simplicity

Yes, but that makes the implementation more complicated... at least as far as I can determine.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ronag ronag Mar 19, 2020

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BridgeAR PTAL

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ronag added 2 commits March 19, 2020 22:55
@ronag
fixup: unhandledRejection
1d0a7a5
@ronag
fixup: faster version
3444617
@ronag ronag added the wip Issues and PRs that are still a work in progress. label Mar 19, 2020
@ronag
fixup: unify
e9e4444
@ronag
ronag force-pushed the pipeline-stdio branch from 327a0a9 to e9e4444 Compare March 19, 2020 23:42
@ronag ronag removed the wip Issues and PRs that are still a work in progress. label Mar 19, 2020
@mcollina

mcollina commented Mar 20, 2020

Copy link
Copy Markdown
Member

What is the performance cost that you are mentioning?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag

ronag commented Mar 20, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

New information from @vweevers which might make this PR outdated.

Ah, my understanding of it was outdated. Since node v10.12.0 (see #23053 and ERR_STDOUT_CLOSE) you can indeed close these streams. Before they used to throw an error, now they allow it but "secretly" don't close the underlying fd.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag ronag added the wip Issues and PRs that are still a work in progress. label Mar 20, 2020
@ronag

ronag commented Mar 20, 2020

Copy link
Copy Markdown
Member Author

I'm putting this as blocked until we have gotten further clarification in the issue:

#32363

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag ronag added the blocked PRs that are blocked by other issues or PRs. label Mar 20, 2020
@ronag
fixup: end stdio
d9d992a
@ronag
ronag force-pushed the pipeline-stdio branch from 8dc9d3a to d9d992a Compare March 20, 2020 09:16
addaleax
addaleax approved these changes Apr 5, 2020
View reviewed changes
mcollina
mcollina approved these changes Apr 5, 2020
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ronag ronag removed blocked PRs that are blocked by other issues or PRs. wip Issues and PRs that are still a work in progress. labels Apr 5, 2020
@nodejs-github-bot

nodejs-github-bot commented Apr 5, 2020

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/30478/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@nodejs-github-bot

nodejs-github-bot commented Apr 5, 2020

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/30479/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag

ronag commented Apr 6, 2020

Copy link
Copy Markdown
Member Author

@addaleax calling end() on stdout seems to have different behavior on Windows.

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/2974/RUN_SUBSET=0,nodes=win10-COMPILED_BY-vs2019/testReport/junit/(root)/test/parallel_test_stream_pipeline_process/

Any thoughts?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag ronag mentioned this pull request Apr 6, 2020
Different behavior between transform function and async generator using pipelines #32363
Closed
@vweevers

vweevers commented Apr 6, 2020

Copy link
Copy Markdown
Contributor
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ '"hello" \r\n'
- 'hello\n'

@ronag That's the behavior of echo "hello" on Windows, you'd get the same result with echo "hello" > test. I suggest removing the double quotes and trimming the result.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag

ronag commented Apr 6, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@ronag That's the behavior of echo "hello" on Windows, you'd get the same result with echo "hello" > test. I suggest removing the double quotes and trimming the result.

So we expect different results on different platforms?

I suggest removing the double quotes and trimming the result.

I don't think that's enough, note the extra space and \r.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ronag added 2 commits April 6, 2020 20:01
@ronag
fixup
1498bb3
@ronag
fixup
2feb611
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@vweevers

vweevers commented Apr 6, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

The default newline on windows is \r\n. The extra space is because echo foo > test adds a space but echo foo> test does not. There are ways around this but normalizing such shell behaviors would not make this node.js test more meaningful.

ronag reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ronag added 2 commits April 6, 2020 20:05
@ronag
fixup
e2ef9c9
@ronag
fixup
51d45eb
@nodejs nodejs deleted a comment from nodejs-github-bot Apr 6, 2020
@nodejs nodejs deleted a comment from nodejs-github-bot Apr 6, 2020
@nodejs-github-bot

nodejs-github-bot commented Apr 6, 2020

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/30510/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag

ronag commented Apr 6, 2020

Copy link
Copy Markdown
Member Author

@mcollina @addaleax PTAL, made test compatible with Windows

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

addaleax
addaleax approved these changes Apr 6, 2020
View reviewed changes

@addaleax addaleax left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
ronag added a commit that referenced this pull request Apr 6, 2020
@ronag
stream: complete pipeline with stdio
98b6b2d
stdio (stderr & stdout) should for compatibility
reasons not be closed/end():ed. However, this
causes pipeline with a stdio destination to
never finish. This commit fixes this issue at
a performance cost.

Refs: #7606

Fixes: #32363

PR-URL: #32373
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
@ronag

ronag commented Apr 6, 2020

Copy link
Copy Markdown
Member Author

Landed in 98b6b2d

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ronag ronag closed this Apr 6, 2020
BethGriggs pushed a commit that referenced this pull request Apr 7, 2020
@ronag @BethGriggs
stream: complete pipeline with stdio
27f6e5d
stdio (stderr & stdout) should for compatibility
reasons not be closed/end():ed. However, this
causes pipeline with a stdio destination to
never finish. This commit fixes this issue at
a performance cost.

Refs: #7606

Fixes: #32363

PR-URL: #32373
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
targos pushed a commit that referenced this pull request Apr 12, 2020
@ronag @targos
stream: complete pipeline with stdio
1c47bba
stdio (stderr & stdout) should for compatibility
reasons not be closed/end():ed. However, this
causes pipeline with a stdio destination to
never finish. This commit fixes this issue at
a performance cost.

Refs: #7606

Fixes: #32363

PR-URL: #32373
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
@targos targos mentioned this pull request Apr 13, 2020
v13.13.0 release proposal #32813
Merged
@targos targos added the backport-blocked-v12.x label Apr 22, 2020
@ronag ronag mentioned this pull request May 4, 2020
stdout not usable after pipeline? #33237
Closed
@ronag ronag mentioned this pull request Jun 26, 2020
Stream.pipeline() does not call the callback on completion #34059
Closed
@mcollina mcollina mentioned this pull request Jul 19, 2021
stream.pipeline does not invoke callback when error happens #39447
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BridgeAR BridgeAR BridgeAR left review comments
@mcollina mcollina mcollina approved these changes
@jasnell jasnell jasnell approved these changes
@addaleax addaleax addaleax approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

stream Issues and PRs related to Node.js streams.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Different behavior between transform function and async generator using pipelines

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@ronag @mcollina @nodejs-github-bot @vweevers @jasnell @addaleax @BridgeAR @targos
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.