GeezerVison

Copy link

Copy Markdown

Contributor

Fishrock123, AndreasMadsen, rvagg, sveisvei, ORESoftware, devsnek, lucasklaassen, and codenirvana reacted with thumbs up emoji Fishrock123, bnb, AndreasMadsen, rvagg, felixfbecker, matteofigus, sveisvei, aqrln, watilde, ORESoftware, and 3 more reacted with hooray emoji

trevnorris force-pushed the async-wrap2 branch from 742ef66 to c235002 Compare March 16, 2017 17:52

trevnorris force-pushed the async-wrap2 branch from c235002 to 73f0041 Compare March 16, 2017 17:59

addaleax reviewed Mar 20, 2017

View reviewed changes

Fishrock123 reviewed Mar 20, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

trevnorris force-pushed the async-wrap2 branch from 73f0041 to 435dee8 Compare March 20, 2017 23:42

not ok 2 parallel/test-async-wrap-getasyncid
  ---
  duration_ms: 0.186
  severity: fail
  stack: |-
    (node:6548) [DEP0064] DeprecationWarning: tls.createSecurePair() is deprecated. Please use tls.Socket instead.
    c:\workspace\node-test-binary-windows\RUN_SUBSET\2\VS_VERSION\vs2015\label\win2016\test\parallel\test-async-wrap-getasyncid.js:176
          throw new Error(`write failed: ${process.binding('uv').errname(err)}`);
          ^
    
    Error: write failed: EPIPE
        at TCPConnectWrap.req.oncomplete.common.mustCall (c:\workspace\node-test-binary-windows\RUN_SUBSET\2\VS_VERSION\vs2015\label\win2016\test\parallel\test-async-wrap-getasyncid.js:176:13)
        at TCPConnectWrap.oncomplete (c:\workspace\node-test-binary-windows\RUN_SUBSET\2\VS_VERSION\vs2015\label\win2016\test\common.js:517:15)

trevnorris force-pushed the async-wrap2 branch from 435dee8 to 9f4c893 Compare March 23, 2017 22:20

trevnorris force-pushed the async-wrap2 branch 2 times, most recently from b39e7b1 to 8a962d2 Compare March 27, 2017 23:24

addaleax reviewed Mar 27, 2017

View reviewed changes

The number of ids is limited to 2^53-1 regardless of whether an int64_t
is used or not because JS is limited to a double. So to make conversion
simpler, track ids internally as a double. This will also make life
simpler when this is eventually exposed to JS via a Float64Array.

Rename AsyncWrap::get_uid() to AsyncWrap::get_id().

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

RandomBytes and PBKDF2 were using the same "generic" ObjectTemplate for
construction. Instead create one for each that is properly named.

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Instead of wrapping several providers into PROVIDER_CRYPTO, have them
all be named after their class. Rename other providers to also match
their class names. With the exception of Parser. Which is actually
HTTPParser.

Add PROVIDER_LENGTH to make better checks in WrapperInfo().

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Allow handles to retrieve their own uid's by adding a new method on the
FunctionTemplates. Implementation of these into all other classes will
come in a future commit.

Add the method AsyncWrap::GetAsyncId() to all inheriting class objects
so the uid of the handle can be retrieved from JS.

In all applicable locations, run ClearWrap() on the object holding the
pointer so that it never points to invalid memory and make sure Wrap()
is always run so the class pointer is correctly attached to the object
and can be retrieved so GetAsyncId() can be run.

In many places a class instance was not removing its own pointer from
object() in the destructor. This left an invalid pointer in the JS
object that could cause the application to segfault under certain
conditions.

Remove ClearWrap() from ReqWrap for continuity. The ReqWrap constructor
was not the one to call Wrap(), so it shouldn't be the one to call
ClearWrap().

Wrap() has been added to all constructors that inherit from AsyncWrap.
Normally it's the child most class. Except in the case of HandleWrap.
Which must be the constructor that runs Wrap() because the class pointer
is retrieved for certain calls and because other child classes have
multiple inheritance to pointer to the HandleWrap needs to be stored.

ClearWrap() has been placed in all FunctionTemplate constructors so that
no random values are returned when running getAsyncId(). ClearWrap() has
also been placed in all class destructors, except in those that use
MakeWeak() because the destructor will run during GC. Making the
object() inaccessible.

It could be simplified to where AsyncWrap sets the internal pointer,
then if an inheriting class needs one of it's own it could set it again.
But the inverse would need to be true also, where AsyncWrap then also
runs ClearWeak. Unforunately because some of the handles are cleaned up
during GC that's impossible. Also in the case of ReqWrap it runs Reset()
in the destructor, making the object() inaccessible. Meaning,
ClearWrap() must be run by the class that runs Wrap(). There's currently
no generalized way of taking care of this across all instances of
AsyncWrap.

I'd prefer that there be checks in there for these things, but haven't
found a way to place them that wouldn't be just as unreliable.

Add test that checks all resources that can run getAsyncId(). Would like
a way to enforce that any new classes that can also run getAsyncId() are
tested, but don't have one.

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Changes in the native code for the upcoming async_hooks module. These
have been separated to help with review and testing.

Changes include:

* Introduce an async id stack that tracks recursive calls into async
  execution contexts. For performance reasons the id stack is held as a
  double* and assigned to a Float64Array. If the stack grows too large
  it is then placed in it's own stack and replaced with a new double*.
  This should accommodate arbitrarily large stacks.

  I'm not especially happy with the complexity involved with this async
  id stack, but it's also the fastest and most full proof way of
  handling it that I have found.

* Add helper functions in Environment and AsyncWrap to work with the
  async id stack.

* Add AsyncWrap::Reset() to allow AsyncWrap instances that have been
  placed in a resource pool, instead of being released, to be
  reinitialized. AsyncWrap::AsyncWrap() also now uses Reset() for
  initialization.

* AsyncWrap* parent no longer needs to be passed via the constructor.

* Introduce Environment::AsyncHooks class to contain the needed native
  functionality. This includes the pointer to the async id stack, and
  array of v8::Eternal<v8::String>'s that hold the names of all
  providers, mechanisms for storing/retrieving the trigger id, etc.

* Introduce Environment::AsyncHooks::ExecScope as a way to track the
  current id and trigger id of function execution via RAII.

* If the user passes --abort-on-uncaught-exception then instead of
  throwing the application will print a stack trace and abort.

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Fill this commit messsage with more details about the change once all
changes are rebased.

* Add lib/async_hooks.js

* Add JS methods to AsyncWrap for handling the async id stack

* Introduce AsyncReset() so that JS functions can reset the id and again
  trigger the init hooks, allow AsyncWrap::Reset() to be called from JS
  via asyncReset().

* Add env variable to test additional things in test/common.js

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Implement async_hooks support in the following:

* fatalException handler
* process.nextTick
* Timers
* net/dgram/http

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Async wrap providers tested:

- crypto.randomBytes
- crypto.pbkdf2
- fs event wrap
- fsreqwrap access
- fsreqwrap readFile
- getaddrinforeq wrap
- getnameinforeq wrap
- pipe connect wrap
- query wrap
- pipewrap
- processwrap
- shutdown wrap
- tcpwrap
- udpwrap
- send wrap
- detailed signal wrap
- statwatcher
- timerwrap via setTimeout
- timerwrap via setInterval
- for Immediate
- http parser request
- http parser response
- connection via ssl server
- tls wrap
- write wrap
- ttywrap via readstream
- ttywrap via wriream
- zctx via zlib binding deflate

Embedder API:

-  async-event tests
  - one test looks at the happy paths
  - another ensures that in cases of events emitted in an order that
  doesn't make sense, the order is enforced by async hooks throwing a
  meaningful error
  - embedder enforcement tests are split up since async hook stack
  corruption now the process
  - therefore we launch a child and check for error output of the offending code

Additional tests:

- tests that show that we can enable/disable hooks inside their lifetime
events
- tests that verify the graph of resources triggering the creation of
other resources

Test Helpers:

- init-hooks:
  - returns one collector instance
  - when created an async hook is created and the lifetime events are
  registered to call the appropriate collector functions
  - the collector also exposes `enable` and `disable` functions which call
  through to the async hook

- hook checks:
  - checks invocations of life time hooks against the actual invocations
  that were collected
  - in some cases like `destroy` a min/max range of invocations can be
  supplied since in these cases the exact number is non-deterministic

- verify graph:
  - verifies the triggerIds of specific async resources are as expected,
  i.e. the creation of resources was triggered by the resource we expect
  - includes a printGraph function to generate easily readable test
  input for verify graph
  - both functions prune TickObjects to create less brittle and easier
  to understand tests

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

PR-URL: #12892
Ref: #11883
Ref: #8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Also add checks in lib/tty.js and tests.

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

AsyncWrap will be going through many changes, and the old API will no
longer be used. So remove those tests that will no longer be useful.
They may be added back later using the new API, once fully implemented.

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

The number of ids is limited to 2^53-1 regardless of whether an int64_t
is used or not because JS is limited to a double. So to make conversion
simpler, track ids internally as a double. This will also make life
simpler when this is eventually exposed to JS via a Float64Array.

Rename AsyncWrap::get_uid() to AsyncWrap::get_id().

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

RandomBytes and PBKDF2 were using the same "generic" ObjectTemplate for
construction. Instead create one for each that is properly named.

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Instead of wrapping several providers into PROVIDER_CRYPTO, have them
all be named after their class. Rename other providers to also match
their class names. With the exception of Parser. Which is actually
HTTPParser.

Add PROVIDER_LENGTH to make better checks in WrapperInfo().

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Allow handles to retrieve their own uid's by adding a new method on the
FunctionTemplates. Implementation of these into all other classes will
come in a future commit.

Add the method AsyncWrap::GetAsyncId() to all inheriting class objects
so the uid of the handle can be retrieved from JS.

In all applicable locations, run ClearWrap() on the object holding the
pointer so that it never points to invalid memory and make sure Wrap()
is always run so the class pointer is correctly attached to the object
and can be retrieved so GetAsyncId() can be run.

In many places a class instance was not removing its own pointer from
object() in the destructor. This left an invalid pointer in the JS
object that could cause the application to segfault under certain
conditions.

Remove ClearWrap() from ReqWrap for continuity. The ReqWrap constructor
was not the one to call Wrap(), so it shouldn't be the one to call
ClearWrap().

Wrap() has been added to all constructors that inherit from AsyncWrap.
Normally it's the child most class. Except in the case of HandleWrap.
Which must be the constructor that runs Wrap() because the class pointer
is retrieved for certain calls and because other child classes have
multiple inheritance to pointer to the HandleWrap needs to be stored.

ClearWrap() has been placed in all FunctionTemplate constructors so that
no random values are returned when running getAsyncId(). ClearWrap() has
also been placed in all class destructors, except in those that use
MakeWeak() because the destructor will run during GC. Making the
object() inaccessible.

It could be simplified to where AsyncWrap sets the internal pointer,
then if an inheriting class needs one of it's own it could set it again.
But the inverse would need to be true also, where AsyncWrap then also
runs ClearWeak. Unforunately because some of the handles are cleaned up
during GC that's impossible. Also in the case of ReqWrap it runs Reset()
in the destructor, making the object() inaccessible. Meaning,
ClearWrap() must be run by the class that runs Wrap(). There's currently
no generalized way of taking care of this across all instances of
AsyncWrap.

I'd prefer that there be checks in there for these things, but haven't
found a way to place them that wouldn't be just as unreliable.

Add test that checks all resources that can run getAsyncId(). Would like
a way to enforce that any new classes that can also run getAsyncId() are
tested, but don't have one.

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Changes in the native code for the upcoming async_hooks module. These
have been separated to help with review and testing.

Changes include:

* Introduce an async id stack that tracks recursive calls into async
  execution contexts. For performance reasons the id stack is held as a
  double* and assigned to a Float64Array. If the stack grows too large
  it is then placed in it's own stack and replaced with a new double*.
  This should accommodate arbitrarily large stacks.

  I'm not especially happy with the complexity involved with this async
  id stack, but it's also the fastest and most full proof way of
  handling it that I have found.

* Add helper functions in Environment and AsyncWrap to work with the
  async id stack.

* Add AsyncWrap::Reset() to allow AsyncWrap instances that have been
  placed in a resource pool, instead of being released, to be
  reinitialized. AsyncWrap::AsyncWrap() also now uses Reset() for
  initialization.

* AsyncWrap* parent no longer needs to be passed via the constructor.

* Introduce Environment::AsyncHooks class to contain the needed native
  functionality. This includes the pointer to the async id stack, and
  array of v8::Eternal<v8::String>'s that hold the names of all
  providers, mechanisms for storing/retrieving the trigger id, etc.

* Introduce Environment::AsyncHooks::ExecScope as a way to track the
  current id and trigger id of function execution via RAII.

* If the user passes --abort-on-uncaught-exception then instead of
  throwing the application will print a stack trace and abort.

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Fill this commit messsage with more details about the change once all
changes are rebased.

* Add lib/async_hooks.js

* Add JS methods to AsyncWrap for handling the async id stack

* Introduce AsyncReset() so that JS functions can reset the id and again
  trigger the init hooks, allow AsyncWrap::Reset() to be called from JS
  via asyncReset().

* Add env variable to test additional things in test/common.js

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Implement async_hooks support in the following:

* fatalException handler
* process.nextTick
* Timers
* net/dgram/http

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

Async wrap providers tested:

- crypto.randomBytes
- crypto.pbkdf2
- fs event wrap
- fsreqwrap access
- fsreqwrap readFile
- getaddrinforeq wrap
- getnameinforeq wrap
- pipe connect wrap
- query wrap
- pipewrap
- processwrap
- shutdown wrap
- tcpwrap
- udpwrap
- send wrap
- detailed signal wrap
- statwatcher
- timerwrap via setTimeout
- timerwrap via setInterval
- for Immediate
- http parser request
- http parser response
- connection via ssl server
- tls wrap
- write wrap
- ttywrap via readstream
- ttywrap via wriream
- zctx via zlib binding deflate

Embedder API:

-  async-event tests
  - one test looks at the happy paths
  - another ensures that in cases of events emitted in an order that
  doesn't make sense, the order is enforced by async hooks throwing a
  meaningful error
  - embedder enforcement tests are split up since async hook stack
  corruption now the process
  - therefore we launch a child and check for error output of the offending code

Additional tests:

- tests that show that we can enable/disable hooks inside their lifetime
events
- tests that verify the graph of resources triggering the creation of
other resources

Test Helpers:

- init-hooks:
  - returns one collector instance
  - when created an async hook is created and the lifetime events are
  registered to call the appropriate collector functions
  - the collector also exposes `enable` and `disable` functions which call
  through to the async hook

- hook checks:
  - checks invocations of life time hooks against the actual invocations
  that were collected
  - in some cases like `destroy` a min/max range of invocations can be
  supplied since in these cases the exact number is non-deterministic

- verify graph:
  - verifies the triggerIds of specific async resources are as expected,
  i.e. the creation of resources was triggered by the resource we expect
  - includes a printGraph function to generate easily readable test
  input for verify graph
  - both functions prune TickObjects to create less brittle and easier
  to understand tests

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>

PR-URL: nodejs#12892
Ref: nodejs#11883
Ref: nodejs#8531
Reviewed-By: Andreas Madsen <[email protected]>
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Sam Roberts <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Refael Ackermann <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Jeremiah Senkpiel <[email protected]>