GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — stream: fix end-of-stream for HTTP/2 by addaleax · Pull Request #24926 · nodejs/node

Conversation

@addaleax

@addaleax addaleax commented Dec 9, 2018

Copy link
Copy Markdown
Member

HTTP/2 streams call .end() on themselves from their
.destroy() method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the _writableState.ended property could be set before
the stream emits its 'close' event, and never actually emits the
'finished' event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing 'finish' and 'end' listeners rather than relying on the
.ended properties of the _...State objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit 'close' before 'end' and/or 'finished', so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: #24456

macOS stress test: https://ci.nodejs.org/job/node-stress-single-test/2124/nodes=osx1011/console

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bricss reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@addaleax
stream: fix end-of-stream for HTTP/2
379c878
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: nodejs#24456
@nodejs-github-bot

nodejs-github-bot commented Dec 9, 2018

Copy link
Copy Markdown
Collaborator

@addaleax build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1880/pipeline

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@addaleax addaleax mentioned this pull request Dec 9, 2018
flaky: test-stream-pipeline-http2 #24456
Closed
@Trott

Trott commented Dec 9, 2018

Copy link
Copy Markdown
Member

Fixes the test-stream-pipeline-http2 problem for me locally. 👍 💯 🎉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Trott
Trott approved these changes Dec 9, 2018
View reviewed changes
@addaleax

addaleax commented Dec 9, 2018

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/19361/

/cc @nodejs/streams

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

lpinca
lpinca approved these changes Dec 9, 2018
View reviewed changes
@addaleax addaleax added stream Issues and PRs related to Node.js streams. author ready PRs with CI started, the required approvals, and no outstanding review comments. labels Dec 9, 2018
mcollina
mcollina reviewed Dec 9, 2018
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work! Can you add a test just using streams?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@addaleax

addaleax commented Dec 9, 2018

Copy link
Copy Markdown
Member Author

@mcollina … I did? https://github.com/nodejs/node/pull/24926/files#diff-c5ecccd43a23798a93a3ab920bebee64 should be what you want, right?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mcollina
mcollina approved these changes Dec 9, 2018
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mcollina

mcollina commented Dec 9, 2018

Copy link
Copy Markdown
Member

(my bad)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

fhinkel
fhinkel approved these changes Dec 10, 2018
View reviewed changes
@Trott

Trott commented Dec 10, 2018 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Resume Build CI: https://ci.nodejs.org/job/node-test-pull-request/19366/ ✔️

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Dec 11, 2018

Copy link
Copy Markdown
Member

Rebuild CI is green, I don't know why the git status reports red. This can land.

schermata 2018-12-11 alle 10 00 41

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Trott

Trott commented Dec 11, 2018

Copy link
Copy Markdown
Member

Landed in 83ec33b

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Trott Trott closed this Dec 11, 2018
Trott pushed a commit to Trott/io.js that referenced this pull request Dec 11, 2018
@addaleax @Trott
stream: fix end-of-stream for HTTP/2
83ec33b
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: nodejs#24456

PR-URL: nodejs#24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
@Trott Trott mentioned this pull request Dec 12, 2018
Revert "stream: make .destroy() interact better with write queue" #24905
Closed
2 tasks done
@addaleax
addaleax deleted the http2-pipeline-fix branch December 12, 2018 11:05
BethGriggs pushed a commit that referenced this pull request Dec 17, 2018
@addaleax @BethGriggs
stream: fix end-of-stream for HTTP/2
39af61f
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: #24456

PR-URL: #24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
@BethGriggs BethGriggs mentioned this pull request Dec 18, 2018
V11.5.0 proposal #25102
Merged
@beevelop beevelop mentioned this pull request Dec 18, 2018
2018-12-18 Version 11.5.0 (Current) @BethGriggs beevelop/docker-android-nodejs#272
Closed
refack pushed a commit to refack/node that referenced this pull request Jan 14, 2019
@refack
stream: fix end-of-stream for HTTP/2
c83cf24
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: nodejs#24456

PR-URL: nodejs#24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
BethGriggs pushed a commit that referenced this pull request Feb 12, 2019
@addaleax @BethGriggs
stream: fix end-of-stream for HTTP/2
bb8a65d
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: #24456

PR-URL: #24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
@BethGriggs BethGriggs added the land-on-v10.x label Feb 12, 2019
@BethGriggs BethGriggs mentioned this pull request Feb 12, 2019
v10.15.3 proposal #26063
Merged
BethGriggs pushed a commit that referenced this pull request Feb 20, 2019
@addaleax @BethGriggs
stream: fix end-of-stream for HTTP/2
11287f8
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: #24456

PR-URL: #24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
rvagg pushed a commit that referenced this pull request Feb 28, 2019
@addaleax @rvagg
stream: fix end-of-stream for HTTP/2
a838116
HTTP/2 streams call `.end()` on themselves from their
`.destroy()` method, which might be queued (e.g. due to network
congestion) and not processed before the stream itself is destroyed.

In that case, the `_writableState.ended` property could be set before
the stream emits its `'close'` event, and never actually emits the
`'finished'` event, confusing the end-of-stream implementation so
that it wouldn’t call its callback.

This can be fixed by watching for the end events themselves using the
existing `'finish'` and `'end'` listeners rather than relying on the
`.ended` properties of the `_...State` objects.

These properties still need to be checked to know whether stream
closure was premature – My understanding is that ideally, streams
should not emit `'close'` before `'end'` and/or `'finished'`, so this
might be another bug, but changing this would require modifying tests
and almost certainly be a breaking change.

Fixes: #24456

PR-URL: #24926
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Luigi Pinca <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Franziska Hinkelmann <[email protected]>
BethGriggs added a commit that referenced this pull request Mar 4, 2019
@BethGriggs
2019-03-05, Version 10.15.3 'Dubnium' (LTS)
1a96d83
Notable Changes

* **doc**
  * add antsmartian to collaborators (Anto Aravinth)
    [#24655](#24655)
* **http**
  * fix error check in Execute() (Brian White)
    [#25863](#25863)
* **stream**
  * fix end-of-stream for HTTP/2 (Anna Henningsen)
    [#24926](#24926)

PR-URL: #26063
BethGriggs added a commit that referenced this pull request Mar 5, 2019
@BethGriggs
2019-03-05, Version 10.15.3 'Dubnium' (LTS)
a52c1ea
Notable Changes

* **doc**
  * add antsmartian to collaborators (Anto Aravinth)
    [#24655](#24655)
* **http**
  * fix error check in Execute() (Brian White)
    [#25863](#25863)
* **stream**
  * fix end-of-stream for HTTP/2 (Anna Henningsen)
    [#24926](#24926)

PR-URL: #26063
@ronag

ronag commented Mar 6, 2019 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

@mcollina Do we have any placeholder issue for known cases of "incorrect" event ordering, in particular in relation to 'close'? If not, maybe we should have one, and if we do then maybe we should add this there?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Mar 6, 2019

Copy link
Copy Markdown
Member

@ronag we don't. Open a new one for HTTP/2.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tadjik1 tadjik1 mentioned this pull request Mar 14, 2019
stream: fix end-of-stream for silent .destroy() #26638
Closed
3 tasks done
@omniadonum omniadonum mentioned this pull request Mar 3, 2020
Node 8 deprecation party 🎉 googleapis/google-cloud-node#2972
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcollina mcollina mcollina approved these changes
@Trott Trott Trott approved these changes
@lpinca lpinca lpinca approved these changes
+1 more reviewer
@fhinkel fhinkel fhinkel approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. stream Issues and PRs related to Node.js streams.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

flaky: test-stream-pipeline-http2

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@addaleax @nodejs-github-bot @Trott @mcollina @ronag @fhinkel @lpinca @BethGriggs
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.