GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — doc: update `stream.reduce` concurrency note by rluvaton · Pull Request #47166 · nodejs/node

Conversation

@rluvaton

@rluvaton rluvaton commented Mar 19, 2023

Copy link
Copy Markdown
Member

At first, when I read that note I thought I should do this:

import { Readable } from 'node:stream';

const totalViews = await Readable.from(videosUrl)
  .map((url) => url, { concurrency: 2 })
  .reduce(async (total, url) => {
    const views = await getViewsAsync(url)
    total + views
  }, 0);

console.log(totalViews);

This did not work and does not make any sense as the map items are emitted in the order that got from the readable stream.

So I thought of changing to note to recommend implementing reduce using .forEach with concurrency but then I understand what @benjamingr meant when he wrote that note - because the only way you will need concurrency is if you do some async operation (because sync block the loop - bla bla) and that operation can be parallelized so I updated the note with an example

@benjamingr, hope this is what you meant 😅

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nodejs-github-bot nodejs-github-bot added doc Issues and PRs related to Node.js documentation. stream Issues and PRs related to Node.js streams. labels Mar 19, 2023
@rluvaton
rluvaton force-pushed the update-stream-operator-reduce-docs branch from 2a31082 to 4b92bfd Compare March 19, 2023 21:18
VoltrexKeyva
VoltrexKeyva reviewed Mar 19, 2023
View reviewed changes
Comment thread doc/api/stream.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread doc/api/stream.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

@rluvaton
rluvaton force-pushed the update-stream-operator-reduce-docs branch 2 times, most recently from 2ac4139 to 487f5db Compare March 19, 2023 22:58
@rluvaton
rluvaton requested a review from VoltrexKeyva March 19, 2023 22:58
@rluvaton
doc: update stream.reduce concurrency note
9627c88
@rluvaton
rluvaton force-pushed the update-stream-operator-reduce-docs branch from 487f5db to 9627c88 Compare March 20, 2023 00:17
rluvaton
rluvaton commented Mar 20, 2023
View reviewed changes
Comment thread doc/api/stream.md
Comment on lines 2508 to +2523
```mjs
import { Readable } from 'node:stream';
import { readdir, stat } from 'node:fs/promises';
import { join } from 'node:path';

const ten = await Readable.from([1, 2, 3, 4]).reduce((previous, data) => {
return previous + data;
});
console.log(ten); // 10
const directoryPath = './src';
const filesInDir = await readdir(directoryPath);

const folderSize = await Readable.from(filesInDir)
.reduce(async (totalSize, file) => {
const { size } = await stat(join(directoryPath, file));
return totalSize + size;
}, 0);

console.log(folderSize);
```

@rluvaton rluvaton Mar 20, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated this example as well so the refactor can be more obvious

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mcollina
mcollina approved these changes Mar 21, 2023
View reviewed changes

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Mar 21, 2023
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Mar 21, 2023
@nodejs-github-bot

nodejs-github-bot commented Mar 21, 2023

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/50527/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@rluvaton

rluvaton commented Mar 23, 2023

Copy link
Copy Markdown
Member Author

Hey @VoltrexKeyva , can you please rereview so it can be merged?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

VoltrexKeyva
VoltrexKeyva approved these changes Mar 23, 2023
View reviewed changes
@rluvaton

rluvaton commented Mar 23, 2023

Copy link
Copy Markdown
Member Author

Thanks @VoltrexKeyva , can you merge it?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@VoltrexKeyva VoltrexKeyva added the commit-queue PRs queued for automated landing through the Commit Queue. label Mar 24, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Mar 24, 2023
@nodejs-github-bot
nodejs-github-bot merged commit c588145 into nodejs:main Mar 24, 2023
@nodejs-github-bot

nodejs-github-bot commented Mar 24, 2023

Copy link
Copy Markdown
Collaborator

Landed in c588145

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@rluvaton
rluvaton deleted the update-stream-operator-reduce-docs branch March 24, 2023 14:42
@RafaelGSS RafaelGSS mentioned this pull request Apr 3, 2023
2023-04-18, Version 20.0.0 (Current) #47381
Merged
RafaelGSS pushed a commit that referenced this pull request Apr 5, 2023
@rluvaton @RafaelGSS
doc: update stream.reduce concurrency note
5fe90d7
PR-URL: #47166
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
@RafaelGSS RafaelGSS mentioned this pull request Apr 6, 2023
v19.9.0 proposal #47441
Merged
RafaelGSS pushed a commit that referenced this pull request Apr 6, 2023
@rluvaton @RafaelGSS
doc: update stream.reduce concurrency note
e1578b6
PR-URL: #47166
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Apr 7, 2023
@rluvaton @RafaelGSS
doc: update stream.reduce concurrency note
908798a
PR-URL: #47166
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
danielleadams pushed a commit that referenced this pull request Jul 6, 2023
@rluvaton @danielleadams
doc: update stream.reduce concurrency note
1317577
PR-URL: #47166
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Mohammed Keyvanzadeh <[email protected]>
@danielleadams danielleadams mentioned this pull request Jul 10, 2023
v18.17.0 release proposal #48694
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcollina mcollina mcollina approved these changes
+1 more reviewer
@VoltrexKeyva VoltrexKeyva VoltrexKeyva approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

doc Issues and PRs related to Node.js documentation. stream Issues and PRs related to Node.js streams.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@rluvaton @nodejs-github-bot @mcollina @VoltrexKeyva
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.