GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — stream: 'readable' have precedence over flowing by mcollina · Pull Request #18994 · nodejs/node

Conversation

@mcollina

@mcollina mcollina commented Feb 26, 2018

Copy link
Copy Markdown
Member

In Streams3 the 'readable' event/.read() method had a lower precedence
than the 'data' event that made them impossible to use them together.
This make .resume() a no-op if there is a listener for the
'readable' event, making the stream non-flowing if there is a
'data'  listener.

Fixes: #18058

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

stream, http

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nodejs-github-bot nodejs-github-bot added the stream Issues and PRs related to Node.js streams. label Feb 26, 2018
@mcollina
mcollina force-pushed the readable-and-data branch from c2fb53b to 53ef8f2 Compare February 26, 2018 08:55
@mcollina mcollina added semver-major PRs that contain breaking changes and should be released in the next major version. dont-land-on-v4.x labels Feb 26, 2018
@mcollina mcollina mentioned this pull request Feb 26, 2018
"readable" event behaving inconsistently #18058
Closed
@mcollina

mcollina commented Feb 26, 2018 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/13369/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1308/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mscdex

mscdex commented Feb 26, 2018

Copy link
Copy Markdown
Contributor

Wouldn't removeAllListeners() to be handled as well, since there could be no 'removeListener' event handlers?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Feb 26, 2018

Copy link
Copy Markdown
Member Author

@mscdex you are right. I didn't account for https://github.com/nodejs/node/blob/master/lib/events.js#L324-L336. Do you think I can just listen for 'removeListener' instead? I thought this approach could theoretically be more performing, but it's an edge case anyway.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mafintosh

mafintosh commented Feb 26, 2018

Copy link
Copy Markdown
Member

@mcollina so basically just suspends flowing when readable is set?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Feb 26, 2018

Copy link
Copy Markdown
Member Author

@mafintosh that's the goal yes. It also restores the behavior after 'readable' is removed.
I'm not sure this is something we want to land in this form (or we might want to refactor).
This is the fix for #18058.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jasnell
jasnell approved these changes Feb 26, 2018
View reviewed changes
@mafintosh

mafintosh commented Feb 27, 2018

Copy link
Copy Markdown
Member

@mcollina wondering if we should even fix this. the same behaivor described in the issue would happen if there are two consumers of the readable event anyway (which is what happens in flowing mode). not too much of a fan of implicit event listener side effects

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Feb 27, 2018

Copy link
Copy Markdown
Member Author

I am good with not fixing it. But it’s a discussion to have.

This fixed a major usability problem if you want to have both a on(‘data’) and on(‘readable’)/read() at the same time (with pipe) read() will always return null.

I’m ok if we want to remove the resume side effect when removing ‘readable’, as that might be confusing. However we should really be updating readableListening: this is also a bug without the fix for read() itself.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mafintosh

mafintosh commented Feb 27, 2018

Copy link
Copy Markdown
Member

Removing the removeListener stuff but keeping the other part sounds like a good middleway to me 👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Mar 6, 2018

Copy link
Copy Markdown
Member

@nodejs/streams @mafintosh PTAL

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@BridgeAR

BridgeAR commented Mar 6, 2018

Copy link
Copy Markdown
Member

CI https://ci.nodejs.org/job/node-test-pull-request/13542/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Mar 7, 2018

Copy link
Copy Markdown
Member Author

@nodejs/tsc what do you think?

An alternative approach might be to remove the readableListening variable and just use listenerCount, hopefully it would be fast enough.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina mcollina mentioned this pull request Mar 14, 2018
readline: add support for async iteration #18904
Closed
4 tasks done
vsemozhetbyt
vsemozhetbyt reviewed Mar 14, 2018
View reviewed changes
Comment thread doc/api/stream.md Outdated

@vsemozhetbyt vsemozhetbyt Mar 14, 2018

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nit: missing period.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mafintosh
mafintosh reviewed Mar 14, 2018
View reviewed changes
Comment thread test/parallel/test-stream3-pause-then-read.js Outdated

@mafintosh mafintosh Mar 14, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you mean to add these logs?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mcollina mcollina Mar 14, 2018

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they follow the prevailing pattern in this test file. Without the console.log statements, this is undebuggable.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mafintosh reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@mafintosh

mafintosh commented Mar 14, 2018

Copy link
Copy Markdown
Member

Lots of good tests! LGTM from me 👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina
mcollina force-pushed the readable-and-data branch from 03e66ed to 44fad80 Compare March 14, 2018 21:17
@mcollina

mcollina commented Mar 14, 2018

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/13685/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1331/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina
mcollina requested review from a team and lpinca March 14, 2018 21:28
@mcollina

mcollina commented Mar 14, 2018

Copy link
Copy Markdown
Member Author

Tagging @nodejs/tsc because it is semver-major.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina mcollina added the tsc-agenda Issues and PRs to discuss during Technical Steering Committee meetings. label Apr 4, 2018
lpinca
lpinca reviewed Apr 4, 2018
View reviewed changes
Comment thread doc/api/stream.md Outdated

@lpinca lpinca Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: not sure why this was changed and it doesn't really matter but for consistency I would keep the arrow function.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@mafintosh mafintosh Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lpinca it is using this.read() below

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@lpinca lpinca Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, ignore me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mcollina
mcollina force-pushed the readable-and-data branch from 44fad80 to 8085e2e Compare April 4, 2018 15:58
@mcollina mcollina added this to the 10.0.0 milestone Apr 4, 2018
@mcollina

mcollina commented Apr 4, 2018

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/14051/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Trott Trott removed the tsc-agenda Issues and PRs to discuss during Technical Steering Committee meetings. label Apr 4, 2018
addaleax
addaleax reviewed Apr 4, 2018
View reviewed changes
Comment thread doc/api/stream.md Outdated

@addaleax addaleax Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you avoid the line break and capitalize the sentence?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread doc/api/stream.md Outdated

@addaleax addaleax Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread lib/_stream_readable.js Outdated

@addaleax addaleax Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be good to have chunk printed out here as well

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread lib/_stream_readable.js Outdated

@addaleax addaleax Apr 4, 2018

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change to comment above this line so that it says why we do this, rather than what we do? I assume it’s about the this.resume() call below?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mcollina
stream: 'readable' have precedence over flowing
578a7b9
In Streams3 the 'readable' event/.read() method had a lower precedence
than the `'data'` event that made them impossible to use them together.
This make `.resume()` a no-op if there is a listener for the
`'readable'` event, making the stream non-flowing if there is a
`'data'`  listener.

Fixes: nodejs#18058
@mcollina
mcollina force-pushed the readable-and-data branch from 8085e2e to 578a7b9 Compare April 4, 2018 21:11
@mcollina

mcollina commented Apr 4, 2018

Copy link
Copy Markdown
Member Author

@addaleax PTAL

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

addaleax
addaleax approved these changes Apr 5, 2018
View reviewed changes
@mcollina

mcollina commented Apr 5, 2018

Copy link
Copy Markdown
Member Author

CI: https://ci.nodejs.org/job/node-test-pull-request/14077/
CITGM (master): https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1353/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1354/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina

mcollina commented Apr 6, 2018

Copy link
Copy Markdown
Member Author

Landed as cf5f986

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mcollina mcollina closed this Apr 6, 2018
@mcollina
mcollina deleted the readable-and-data branch April 6, 2018 12:17
mcollina added a commit that referenced this pull request Apr 6, 2018
@mcollina
stream: 'readable' have precedence over flowing
cf5f986
In Streams3 the 'readable' event/.read() method had a lower precedence
than the `'data'` event that made them impossible to use them together.
This make `.resume()` a no-op if there is a listener for the
`'readable'` event, making the stream non-flowing if there is a
`'data'`  listener.

Fixes: #18058

PR-URL: #18994
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
@MylesBorins

MylesBorins commented Apr 11, 2018

Copy link
Copy Markdown
Contributor

Seems like failures in cf5f986#diff-573b8412079b987e160a3fde511d5b9b are showing up in a couple places

In a couple CI runs in #19201 and also in http://github.com/nodejs/node/pull/19924

@mcollina thoughts?

not ok 750 parallel/test-http-readable-data-event
  ---
  duration_ms: 0.725
  severity: fail
  stack: |-
    assert.js:79
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: 'Hello World!Hello again later!' strictEqual 'Hello World!'
        at IncomingMessage.res.on.common.mustCall (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/parallel/test-http-readable-data-event.js:43:14)
        at IncomingMessage.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/common/index.js:467:15)
        at IncomingMessage.emit (events.js:182:13)
        at IncomingMessage.Readable.read (_stream_readable.js:489:10)
        at IncomingMessage.res.on.common.mustCall (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/parallel/test-http-readable-data-event.js:36:20)
        at IncomingMessage.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/common/index.js:467:15)
        at IncomingMessage.emit (events.js:182:13)
        at emitReadable_ (_stream_readable.js:537:12)
        at process._tickCallback (internal/process/next_tick.js:174:19)
  ...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Apr 11, 2018
assert: fix actual & expected input #19925
Closed
deps: update V8 to 6.6 #19201
Closed
@mcollina

mcollina commented Apr 11, 2018

Copy link
Copy Markdown
Member Author

@MylesBorins We are already tracking it in #19905.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lpinca lpinca mentioned this pull request May 6, 2018
Behavior of readablestreams changed in v10 #20520
Closed
schnittstabil added a commit to schnittstabil/cp-file that referenced this pull request May 11, 2018
@schnittstabil
fix node 10 compatibility
2841594
Caused by nodejs/node#18994
sindresorhus pushed a commit to sindresorhus/copy-file that referenced this pull request May 12, 2018
@schnittstabil @sindresorhus
Fix Node.js 10 compatibility
4957feb
Caused by nodejs/node#18994

Closes #26
@mcollina mcollina mentioned this pull request Jun 2, 2018
Node read stream hangs on specific file, base64 encoding #19862
Closed
@mcollina mcollina mentioned this pull request Jul 5, 2018
http request 'end' event not fired #21398
Closed
@lundibundi lundibundi mentioned this pull request Jul 30, 2018
ReadStream paused and can't resume, when listened readable event #22032
Closed
@mcollina mcollina mentioned this pull request Aug 9, 2018
stream: restore flow if there are 'data' handlers after once('readable') #22209
Closed
4 tasks done
@mcollina mcollina mentioned this pull request Aug 21, 2018
fs.createWriteStream does not fire close event #21122
Closed
@snakamura snakamura mentioned this pull request Nov 10, 2018
stream.resume() doesn't make the stream flowing after removing readable listener #24281
Closed
@arantes555 arantes555 mentioned this pull request Dec 7, 2018
Flow problem with node >= 10.10 juliangruber/multipipe#48
Closed
@mcollina mcollina mentioned this pull request Nov 11, 2021
stream: adjust src hwm when pipelining #40751
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lpinca lpinca lpinca left review comments
@jasnell jasnell jasnell approved these changes
@addaleax addaleax addaleax approved these changes
+2 more reviewers
@mafintosh mafintosh mafintosh approved these changes
@vsemozhetbyt vsemozhetbyt vsemozhetbyt left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

semver-major PRs that contain breaking changes and should be released in the next major version. stream Issues and PRs related to Node.js streams.

Projects

None yet

Milestone

10.0.0

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

11 participants

@mcollina @mscdex @mafintosh @BridgeAR @MylesBorins @jasnell @addaleax @lpinca @vsemozhetbyt @Trott @nodejs-github-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.