GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — src: implement DataQueue and non-memory resident Blob by jasnell · Pull Request #45258 · nodejs/node

Conversation

jasnell commented Oct 31, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

This is ~~the start of~~ work ~~being~~ collaboratively done by myself and @flakey5 to update the implementation of Blob to support non-memory resident data sources such as files as well as enable proper streaming support.

~~It is a work in progress and remain draft status until it is ready to go. Opening it for transparency.~~

Some background:

Currently, Blob objects consist solely of memory-resident data, backed by a collection of v8::BackingStore instances. While this behavior is correct standard/spec defined behavior, it is extremely inefficient. To support other use cases of Blob, we want to be able to efficiently and effectively support non-memory resident data sources such as files. To do so will require a number of key changes on the internals of the Blob class.

Further, the current implementation of Blob suffers from an overly simplistic streaming model where the entire Blob is first read into a single ArrayBuffer before passing the data on to the stream API. Effectively meaning Blobs are never actually streamed.

While going through this evaluation, we came to realization that many of the underlying requirements for Blob are (surprisingly) shared by the internal data management requirements for the QUIC implementation that is in progress, and can enable a range of other use cases also, so we have decided to take an approach that addresses those multiple use cases.

~~When complete, this PR is going to do two things:~~

This PR does three things:

Introduces a new C++ data structure called DataQueue which acts as a sequence of memory-resident and non-memory-resident data sources that can be consumed as a single logical stream of data. This is designed to meet the needs of both Blob and QUIC initially (the bits relevant to QUIC will come later)
Update the current Blob implementation to use DataQueue internally in support of the existing all-memory-resident model.
Provides an API for acquiring a File-backed Blob that will support efficient, non-memory resident use cases.

This PR is far from complete. I would strongly recommend that you hold off performing any review on it at all until after we flip the bit to mark it ready for review. We are opening this draft PR now purely for the sake of transparency so that folks can be aware of what we are working on.

I also want to avoid any and all bikeshedding on the design until we at least have the updated Blob implementation ready to go, and many of the design decisions made will likely not be clear until that concrete case is implemented.

Specifically for Blob, this PR ~~will enable~~ enables:

Proper streaming support. No longer will the Blob data be collected into a single ArrayBuffer and then passed on to the stream, but data will flow into the stream as one would actually expect.
File-backed Blob. The ability to acquire a Blob that is backed by an on-disc file. As well as the ability to use such Blob instances efficiently within other Blob instances, e.g. new Blob(['string', fileBackedBlob])

For QUIC, eventually the DataQueue implemented here will replace the Stream::Queue mechanism implemented in that PR. DataQueue will not completely replace that mechanism but covers a sizable chunk of it, which should further simplify and reduce the size of that rather complex bit of work.

As for other use cases of DataQueue... we'll address those later, but we do have a range of cases in mind (including the recent discussions around Node.js providing an efficient http static file server).

/cc @mcollina (who prompted this work due to Undici and Fetch use case requirements)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

theanarkh, bricss, benjamingr, jimmywarting, and styfle reacted with thumbs up emoji

tonivj5, mcollina, bricss, RaisinTen, jimmywarting, and metcoder95 reacted with heart emoji

ronag, KhafraDev, flakey5, bricss, RaisinTen, jimmywarting, gfortaine, styfle, PositiveGrid-it, and hikarunoryoma reacted with rocket emoji

bricss, devsnek, and metcoder95 reacted with eyes emoji

All reactions

👍 5 reactions
❤️ 6 reactions
🚀 10 reactions
👀 3 reactions

nodejs-github-bot added build

Issues and PRs related to Node.js builds or CI infrastructure.

c++

Issues and PRs that require attention from people who are familiar with C++.

needs-ci PRs that need a full CI run. labels

Oct 31, 2022

nodejs deleted a comment from nodejs-github-bot

Oct 31, 2022

mcollina reviewed

Nov 1, 2022

View reviewed changes

Comment thread

src/dataqueue/queue.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

src/dataqueue/queue.h Resolved

Uh oh!

There was an error while loading. Please reload this page.

Trott force-pushed the main branch from 2d76238 to ca3ed36 Compare

November 12, 2022 01:49

jasnell force-pushed the dataqueue branch from 357a86b to 89b638f Compare

November 19, 2022 19:04

jasnell force-pushed the dataqueue branch from 9d028aa to 0dcc1a1 Compare

December 17, 2022 21:11

jasnell marked this pull request as ready for review

December 17, 2022 21:59

jasnell commented Dec 17, 2022

Copy link

Copy Markdown

Member Author

@mcollina ... this is ready for review!

flakey5 reacted with hooray emoji

All reactions

🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mcollina reviewed

Dec 18, 2022

View reviewed changes

Comment thread

test/parallel/test-blob-file-backed.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

doc/api/fs.md Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

test/parallel/test-blob-file-backed.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

mcollina commented Dec 18, 2022

Copy link

Copy Markdown

Member

cc @KhafraDev

KhafraDev reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mcollina approved these changes

Dec 18, 2022

View reviewed changes

mcollina left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This comment was marked as outdated.

github-actions Bot mentioned this pull request

Dec 19, 2022

CI Reliability 2022-12-19 nodejs/reliability#462

Open

14 tasks

This comment was marked as outdated.

jasnell force-pushed the dataqueue branch 2 times, most recently from 9b037f1 to 18ba98e Compare

December 19, 2022 23:56

github-actions Bot mentioned this pull request

Dec 20, 2022

CI Reliability 2022-12-20 nodejs/reliability#463

Open

14 tasks

This comment was marked as outdated.

jasnell added semver-minor

PRs that contain new features and should be released in the next minor version.

experimental Issues and PRs related to experimental features. labels

Dec 20, 2022

nodejs-github-bot commented Dec 20, 2022

Copy link

Copy Markdown

Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/48655/

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

github-actions Bot mentioned this pull request

Dec 21, 2022

CI Reliability 2022-12-21 nodejs/reliability#464

Open

12 tasks

addaleax previously requested changes

Dec 21, 2022

There was an error while loading. Please reload this page.

github-actions Bot mentioned this pull request

Dec 22, 2022

CI Reliability 2022-12-22 nodejs/reliability#465

Open

14 tasks

jasnell commented Dec 22, 2022

Copy link

Copy Markdown

Member Author

Moving this back to Draft status. Trying to work through @addaleax's feedback and wanted to make it so that the FdEntry Reader is using FileHandle as a StreamBase but for some reason I'm having a difficult time getting it to work to perform parallel reads of the same file with two separate fds. Will move back out of draft status once I've got it working.

addaleax reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jasnell marked this pull request as draft

December 22, 2022 21:02

github-actions Bot mentioned this pull request

Dec 23, 2022

CI Reliability 2022-12-23 nodejs/reliability#466

Open

13 tasks

juanarbol commented Mar 1, 2023

Copy link

Copy Markdown

Member

Would you mind to backporting this to v18.x?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

targos pushed a commit that referenced this pull request

Mar 13, 2023


          src: implement DataQueue

9b6270a

See documentation in dataqueue/queue.h for details

Co-authored-by: flakey5 <[email protected]>
PR-URL: #45258
Reviewed-By: Matteo Collina <[email protected]>

targos pushed a commit that referenced this pull request

Mar 13, 2023


          src: update Blob implementation to use DataQueue / File-backed Blobs

36f36b9

Co-authored-by: flakey5 <[email protected]>
PR-URL: #45258
Reviewed-By: Matteo Collina <[email protected]>

targos pushed a commit that referenced this pull request

Mar 13, 2023


          src, lib: fixup lint and format issues for DataQueue/Blob

d48ed95

Co-authored-by: flakey5 <[email protected]>
PR-URL: #45258
Reviewed-By: Matteo Collina <[email protected]>

targos mentioned this pull request

Mar 14, 2023

v19.8.0 release proposal #47087

Merged

targos added a commit that referenced this pull request

Mar 14, 2023


          2023-03-14, Version 19.8.0 (Current)

4221dce

Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47087

targos added a commit that referenced this pull request

Mar 14, 2023


          2023-03-14, Version 19.8.0 (Current)

115c9ac

Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47087

jimmywarting commented Mar 18, 2023

Copy link

Copy Markdown

I notice that blobs transfered over workers aren't readable at all.

it only applies to blobs coming from fs.openAsBlob
in-memory constructed blobs works fine.

mini repro:

import { Worker } from 'worker_threads'
new Worker('./worker.js').once('message', blob => blob.text())

// worker.js
import { parentPort } from 'node:worker_threads'
import { openAsBlob } from 'node:fs'

const blob = await openAsBlob(import.meta.url.slice(7)) // file:// doesn't work
parentPort.postMessage(blob)

I would assume that if you also closed the worker it would still work just fine to read the blob, even if it came from another thread...

Error stacktrace

FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope
 1: 0x104e1df7c node::Abort() [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 2: 0x104e1f33c node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 3: 0x104f7a064 v8::Utils::ReportApiFailure(char const*, char const*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 4: 0x1050cb130 v8::internal::HandleScope::Extend(v8::internal::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 5: 0x104f7bf04 v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 6: 0x104f98978 v8::ObjectTemplate::NewInstance(v8::Local<v8::Context>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 7: 0x104e241dc node::fs::FileHandle::New(node::fs::BindingData*, int, v8::Local<v8::Object>, std::__1::optional<long long>, std::__1::optional<long long>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 8: 0x104d869ec node::(anonymous namespace)::FdEntry::get_reader() [/opt/homebrew/Cellar/node/19.8.1/bin/node]
 9: 0x104d847b8 node::(anonymous namespace)::IdempotentDataQueueReader::Pull(std::__1::function<void (int, node::DataQueue::Vec const*, unsigned long, std::__1::function<void (unsigned long)>)>, int, node::DataQueue::Vec*, unsigned long, unsigned long) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
10: 0x104d86518 node::(anonymous namespace)::DataQueueEntry::ReaderImpl::Pull(std::__1::function<void (int, node::DataQueue::Vec const*, unsigned long, std::__1::function<void (unsigned long)>)>, int, node::DataQueue::Vec*, unsigned long, unsigned long) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
11: 0x104d84928 node::(anonymous namespace)::IdempotentDataQueueReader::Pull(std::__1::function<void (int, node::DataQueue::Vec const*, unsigned long, std::__1::function<void (unsigned long)>)>, int, node::DataQueue::Vec*, unsigned long, unsigned long) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
12: 0x104defcf8 node::Blob::Reader::Pull(v8::FunctionCallbackInfo<v8::Value> const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
13: 0x104fd8a58 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
14: 0x104fd8858 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, unsigned long*, int) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
15: 0x104fd82b8 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
16: 0x104c733ec Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/opt/homebrew/Cellar/node/19.8.1/bin/node]
17: 0x104bf0064 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
18: 0x104bf0064 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
19: 0x104bf0064 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
20: 0x104bf0064 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
21: 0x104c28ab0 Builtins_GeneratorPrototypeNext [/opt/homebrew/Cellar/node/19.8.1/bin/node]
22: 0x104bee4f0 Builtins_JSEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
23: 0x104bee184 Builtins_JSEntry [/opt/homebrew/Cellar/node/19.8.1/bin/node]
24: 0x10509aa90 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
25: 0x10509b1a8 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
26: 0x1053a0c8c v8::internal::SourceTextModule::ExecuteModule(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
27: 0x1053a067c v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>, v8::internal::ZoneForwardList<v8::internal::Handle<v8::internal::SourceTextModule> >*, unsigned int*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
28: 0x1053a03e0 v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>, v8::internal::ZoneForwardList<v8::internal::Handle<v8::internal::SourceTextModule> >*, unsigned int*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
29: 0x10539ff6c v8::internal::SourceTextModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
30: 0x105355240 v8::internal::Module::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Module>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
31: 0x104f849f8 v8::Module::Evaluate(v8::Local<v8::Context>) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
32: 0x104de085c node::loader::ModuleWrap::Evaluate(v8::FunctionCallbackInfo<v8::Value> const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
33: 0x104fd8a58 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
34: 0x104fd8858 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, unsigned long*, int) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
35: 0x104fd82b8 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
36: 0x104c733ec Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/opt/homebrew/Cellar/node/19.8.1/bin/node]
37: 0x104bf0064 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/19.8.1/bin/node]
38: 0x104c268b4 Builtins_AsyncFunctionAwaitResolveClosure [/opt/homebrew/Cellar/node/19.8.1/bin/node]
39: 0x104cc6e38 Builtins_PromiseFulfillReactionJob [/opt/homebrew/Cellar/node/19.8.1/bin/node]
40: 0x104c16834 Builtins_RunMicrotasks [/opt/homebrew/Cellar/node/19.8.1/bin/node]
41: 0x104bee3c4 Builtins_JSRunMicrotasksEntry [/opt/homebrew/Cellar/node/19.8.1/bin/node]
42: 0x10509aa60 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
43: 0x10509b1a8 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
44: 0x1050bdc7c v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
45: 0x1050bdaac v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
46: 0x104d50ae8 node::InternalCallbackScope::Close() [/opt/homebrew/Cellar/node/19.8.1/bin/node]
47: 0x104e25864 node::fs::FileHandle::CloseReq::Resolve() [/opt/homebrew/Cellar/node/19.8.1/bin/node]
48: 0x104e26304 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
49: 0x104e182b4 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
50: 0x10784aff8 uv__work_done [/opt/homebrew/Cellar/libuv/1.44.2/lib/libuv.1.dylib]
51: 0x10784e3d0 uv__async_io [/opt/homebrew/Cellar/libuv/1.44.2/lib/libuv.1.dylib]
52: 0x10785e1e0 uv__io_poll [/opt/homebrew/Cellar/libuv/1.44.2/lib/libuv.1.dylib]
53: 0x10784e7d0 uv_run [/opt/homebrew/Cellar/libuv/1.44.2/lib/libuv.1.dylib]
54: 0x104d518a8 node::SpinEventLoopInternal(node::Environment*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
55: 0x104e60068 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
56: 0x104e5fde8 node::NodeMainInstance::Run() [/opt/homebrew/Cellar/node/19.8.1/bin/node]
57: 0x104de6654 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
58: 0x104de67d8 node::Start(int, char**) [/opt/homebrew/Cellar/node/19.8.1/bin/node]
59: 0x1a2cd7e50 start [/usr/lib/dyld]

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jasnell commented Mar 18, 2023

Copy link

Copy Markdown

Member Author

Oops, yeah just a bug. Will take a look in the next day or so.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jimmywarting mentioned this pull request

Mar 20, 2023

Implement File class #39015

Closed

targos commented Mar 25, 2023

Copy link

Copy Markdown

Member

I also noticed that the new method is on fs, not fs.promises, which is weird since it returns a promise.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jimmywarting mentioned this pull request

Mar 31, 2023

Transfered (PostMessaged) disk- blob's aren't readable #47334

Closed

jimmywarting mentioned this pull request

Apr 22, 2023

Make disk- blob transferable #47666

Closed

jimmywarting mentioned this pull request

Jul 24, 2023

Getting Files backed up by the filesystem #37340

Closed

H4ad mentioned this pull request

Sep 21, 2023

Blob.text and Blob.arraybuffer are 10x+ slower on 20.x nodejs/performance#118

Open

LiviaMedeiros mentioned this pull request

Sep 22, 2023

fs: fix fs.openAsBlob, add fs.openAsBlobSync and fsPromises.openAsBlob #49759

Closed

codebytere added a commit to electron/electron that referenced this pull request

Nov 6, 2023


          src: implement DataQueue and non-memory resident Blob

4034a92

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 14, 2023


          src: implement DataQueue and non-memory resident Blob

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 15, 2023


          src: implement DataQueue and non-memory resident Blob

06d29b6

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 16, 2023


          src: implement DataQueue and non-memory resident Blob

4cdbe2b

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 21, 2023


          src: implement DataQueue and non-memory resident Blob

bba962e

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 22, 2023


          src: implement DataQueue and non-memory resident Blob

e9e094f

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 28, 2023


          src: implement DataQueue and non-memory resident Blob

65fc29c

nodejs/node#45258

codebytere added a commit to electron/electron that referenced this pull request

Nov 29, 2023


          src: implement DataQueue and non-memory resident Blob

567a3b3

nodejs/node#45258

jkleinsc pushed a commit to electron/electron that referenced this pull request

Nov 30, 2023


          chore: upgrade to Node.js v20 (#40545)

9c94fd7

* chore: upgrade to Node.js v20

* src: allow embedders to override NODE_MODULE_VERSION

nodejs/node#49279

* src: fix missing trailing ,

nodejs/node#46909

* src,tools: initialize cppgc

nodejs/node#45704

* tools: allow passing absolute path of config.gypi in js2c

nodejs/node#49162

* tools: port js2c.py to C++

nodejs/node#46997

* doc,lib: disambiguate the old term, NativeModule

nodejs/node#45673

* chore: fixup Node.js BSSL tests

* nodejs/node#49492
* nodejs/node#44498

* deps: upgrade to libuv 1.45.0

nodejs/node#48078

* deps: update V8 to 10.7

nodejs/node#44741

* test: use gcUntil() in test-v8-serialize-leak

nodejs/node#49168

* module: make CJS load from ESM loader

nodejs/node#47999

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* chore: address changes to CJS/ESM loading

* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)

* bootstrap: optimize modules loaded in the built-in snapshot

nodejs/node#45849

* test: mark test-runner-output as flaky

nodejs/node#49854

* lib: lazy-load deps in modules/run_main.js

nodejs/node#45849

* url: use private properties for brand check

nodejs/node#46904

* test: refactor `test-node-output-errors`

nodejs/node#48992

* assert: deprecate callTracker

nodejs/node#47740

* src: cast v8::Object::GetInternalField() return value to v8::Value

nodejs/node#48943

* test: adapt test-v8-stats for V8 update

nodejs/node#45230

* tls: ensure TLS Sockets are closed if the underlying wrap closes

nodejs/node#49327

* test: deflake test-tls-socket-close

nodejs/node#49575

* net: fix crash due to simultaneous close/shutdown on JS Stream Sockets

nodejs/node#49400

* net: use asserts in JS Socket Stream to catch races in future

nodejs/node#49400

* lib: fix BroadcastChannel initialization location

nodejs/node#46864

* src: create BaseObject with node::Realm

nodejs/node#44348

* src: implement DataQueue and non-memory resident Blob

nodejs/node#45258

* sea: add support for V8 bytecode-only caching

nodejs/node#48191

* chore: fixup patch indices

* gyp: put filenames in variables

nodejs/node#46965

* build: modify js2c.py into GN executable

* fix: (WIP) handle string replacement of fs -> original-fs

* [v20.x] backport vm-related memory fixes

nodejs/node#49874

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* src: avoid copying string in fs_permission

nodejs/node#47746

* look upon my works ye mighty

and dispair

* chore: patch cleanup

* [api] Remove AllCan Read/Write

https://chromium-review.googlesource.com/c/v8/v8/+/5006387

* fix: missing include for NODE_EXTERN

* chore: fixup patch indices

* fix: fail properly when js2c fails in Node.js

* build: fix js2c root_gen_dir

* fix: lib/fs.js -> lib/original-fs.js

* build: fix original-fs file xforms

* fixup! module: make CJS load from ESM loader

* build: get rid of CppHeap for now

* build: add patch to prevent extra fs lookup on esm load

* build: greatly simplify js2c modifications

Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c

* chore: update to handle moved internal/modules/helpers file

* test: update @types/node test

* feat: enable preventing cppgc heap creation

* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler

* fix: no cppgc initialization in the renderer

* gyp: put filenames in variables

nodejs/node#46965

* test: disable single executable tests

* fix: nan tests failing on node headers missing file

* tls,http2: send fatal alert on ALPN mismatch

nodejs/node#44031

* test: disable snapshot tests

* nodejs/node#47887
* nodejs/node#49684
* nodejs/node#44193

* build: use deps/v8 for v8/tools

Node.js hard depends on these in their builtins

* test: fix edge snapshot stack traces

nodejs/node#49659

* build: remove js2c //base dep

* build: use electron_js2c_toolchain to build node_js2c

* fix: don't create SafeSet outside packageResolve

Fixes failure in parallel/test-require-delete-array-iterator:

=== release test-require-delete-array-iterator ===
Path: parallel/test-require-delete-array-iterator
node:internal/per_context/primordials:426
    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor
                     ^

TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))
    at new Set (<anonymous>)
    at new SafeSet (node:internal/per_context/primordials:426:22)

* fix: failing crashReporter tests on Linux

These were failing because our change from node::InitializeNodeWithArgs to
node::InitializeOncePerProcess meant that we now inadvertently called
PlatformInit, which reset signal handling. This meant that our intentional
crash function ElectronBindings::Crash no longer worked and the renderer process
no longer crashed when process.crash() was called. We don't want to use Node.js'
default signal handling in the renderer process, so we disable it by passing
kNoDefaultSignalHandling to node::InitializeOncePerProcess.

* build: only create cppgc heap on non-32 bit platforms

* chore: clean up util:CompileAndCall

* src: fix compatility with upcoming V8 12.1 APIs

nodejs/node#50709

* fix: use thread_local BuiltinLoader

* chore: fixup v8 patch indices

---------

Co-authored-by: Keeley Hammond <[email protected]>
Co-authored-by: Samuel Attard <[email protected]>

MrHuangJser pushed a commit to MrHuangJser/electron that referenced this pull request

Dec 11, 2023


          chore: upgrade to Node.js v20 (electron#40545)

34651d1

* chore: upgrade to Node.js v20

* src: allow embedders to override NODE_MODULE_VERSION

nodejs/node#49279

* src: fix missing trailing ,

nodejs/node#46909

* src,tools: initialize cppgc

nodejs/node#45704

* tools: allow passing absolute path of config.gypi in js2c

nodejs/node#49162

* tools: port js2c.py to C++

nodejs/node#46997

* doc,lib: disambiguate the old term, NativeModule

nodejs/node#45673

* chore: fixup Node.js BSSL tests

* nodejs/node#49492
* nodejs/node#44498

* deps: upgrade to libuv 1.45.0

nodejs/node#48078

* deps: update V8 to 10.7

nodejs/node#44741

* test: use gcUntil() in test-v8-serialize-leak

nodejs/node#49168

* module: make CJS load from ESM loader

nodejs/node#47999

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* chore: address changes to CJS/ESM loading

* module: make CJS load from ESM loader (nodejs/node#47999)
* lib: improve esm resolve performance (nodejs/node#46652)

* bootstrap: optimize modules loaded in the built-in snapshot

nodejs/node#45849

* test: mark test-runner-output as flaky

nodejs/node#49854

* lib: lazy-load deps in modules/run_main.js

nodejs/node#45849

* url: use private properties for brand check

nodejs/node#46904

* test: refactor `test-node-output-errors`

nodejs/node#48992

* assert: deprecate callTracker

nodejs/node#47740

* src: cast v8::Object::GetInternalField() return value to v8::Value

nodejs/node#48943

* test: adapt test-v8-stats for V8 update

nodejs/node#45230

* tls: ensure TLS Sockets are closed if the underlying wrap closes

nodejs/node#49327

* test: deflake test-tls-socket-close

nodejs/node#49575

* net: fix crash due to simultaneous close/shutdown on JS Stream Sockets

nodejs/node#49400

* net: use asserts in JS Socket Stream to catch races in future

nodejs/node#49400

* lib: fix BroadcastChannel initialization location

nodejs/node#46864

* src: create BaseObject with node::Realm

nodejs/node#44348

* src: implement DataQueue and non-memory resident Blob

nodejs/node#45258

* sea: add support for V8 bytecode-only caching

nodejs/node#48191

* chore: fixup patch indices

* gyp: put filenames in variables

nodejs/node#46965

* build: modify js2c.py into GN executable

* fix: (WIP) handle string replacement of fs -> original-fs

* [v20.x] backport vm-related memory fixes

nodejs/node#49874

* src: make BuiltinLoader threadsafe and non-global

nodejs/node#45942

* src: avoid copying string in fs_permission

nodejs/node#47746

* look upon my works ye mighty

and dispair

* chore: patch cleanup

* [api] Remove AllCan Read/Write

https://chromium-review.googlesource.com/c/v8/v8/+/5006387

* fix: missing include for NODE_EXTERN

* chore: fixup patch indices

* fix: fail properly when js2c fails in Node.js

* build: fix js2c root_gen_dir

* fix: lib/fs.js -> lib/original-fs.js

* build: fix original-fs file xforms

* fixup! module: make CJS load from ESM loader

* build: get rid of CppHeap for now

* build: add patch to prevent extra fs lookup on esm load

* build: greatly simplify js2c modifications

Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c

* chore: update to handle moved internal/modules/helpers file

* test: update @types/node test

* feat: enable preventing cppgc heap creation

* feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler

* fix: no cppgc initialization in the renderer

* gyp: put filenames in variables

nodejs/node#46965

* test: disable single executable tests

* fix: nan tests failing on node headers missing file

* tls,http2: send fatal alert on ALPN mismatch

nodejs/node#44031

* test: disable snapshot tests

* nodejs/node#47887
* nodejs/node#49684
* nodejs/node#44193

* build: use deps/v8 for v8/tools

Node.js hard depends on these in their builtins

* test: fix edge snapshot stack traces

nodejs/node#49659

* build: remove js2c //base dep

* build: use electron_js2c_toolchain to build node_js2c

* fix: don't create SafeSet outside packageResolve

Fixes failure in parallel/test-require-delete-array-iterator:

=== release test-require-delete-array-iterator ===
Path: parallel/test-require-delete-array-iterator
node:internal/per_context/primordials:426
    constructor(i) { super(i); } // eslint-disable-line no-useless-constructor
                     ^

TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator))
    at new Set (<anonymous>)
    at new SafeSet (node:internal/per_context/primordials:426:22)

* fix: failing crashReporter tests on Linux

These were failing because our change from node::InitializeNodeWithArgs to
node::InitializeOncePerProcess meant that we now inadvertently called
PlatformInit, which reset signal handling. This meant that our intentional
crash function ElectronBindings::Crash no longer worked and the renderer process
no longer crashed when process.crash() was called. We don't want to use Node.js'
default signal handling in the renderer process, so we disable it by passing
kNoDefaultSignalHandling to node::InitializeOncePerProcess.

* build: only create cppgc heap on non-32 bit platforms

* chore: clean up util:CompileAndCall

* src: fix compatility with upcoming V8 12.1 APIs

nodejs/node#50709

* fix: use thread_local BuiltinLoader

* chore: fixup v8 patch indices

---------

Co-authored-by: Keeley Hammond <[email protected]>
Co-authored-by: Samuel Attard <[email protected]>

vpanta mentioned this pull request

Mar 25, 2024

[Bug]: js_binary can't find transitively-dependent modules (Node 19.8+) aspect-build/rules_js#1546

Closed

dario-piotrowicz mentioned this pull request

May 13, 2025

typings: remove blob extra typings #58305

Closed

Reviewers

addaleax

addaleax left review comments

mcollina

mcollina approved these changes

+1 more reviewer

jimmywarting

jimmywarting left review comments

Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

build

Issues and PRs related to Node.js builds or CI infrastructure.

c++

Issues and PRs that require attention from people who are familiar with C++.

experimental

Issues and PRs related to experimental features.

needs-ci

PRs that need a full CI run.

semver-minor

PRs that contain new features and should be released in the next minor version.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

jasnell commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasnell commented Dec 17, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcollina commented Dec 18, 2022

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

nodejs-github-bot commented Dec 20, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasnell commented Dec 22, 2022

Uh oh!

juanarbol commented Mar 1, 2023

Uh oh!

jimmywarting commented Mar 18, 2023

Uh oh!

jasnell commented Mar 18, 2023

Uh oh!

targos commented Mar 25, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jasnell commented Oct 31, 2022 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.