GeezerVison is showing github.com — events: improve performance by ~120% by anonrig · Pull Request #44627 · nodejs/node
Conversation
Object.create(null) is 20x slower on Node 14, and 10x slower on Node 18. This is mainly related to the bug I've opened: https://bugs.chromium.org/p/v8/issues/detail?id=13273#c2
V8 is super optimized for new Cls(), but not for Object.create(null)
events/ee-add-remove.js n=1000000 removeListener=0 newListener=0 *** 18.89 % ±0.88% ±1.17% ±1.52%
events/ee-add-remove.js n=1000000 removeListener=0 newListener=1 *** 13.76 % ±1.23% ±1.64% ±2.15%
events/ee-add-remove.js n=1000000 removeListener=1 newListener=0 *** 13.46 % ±1.08% ±1.44% ±1.89%
events/ee-add-remove.js n=1000000 removeListener=1 newListener=1 *** 3.88 % ±1.05% ±1.40% ±1.84%
events/ee-emit.js listeners=1 argc=0 n=2000000 *** 116.87 % ±2.80% ±3.77% ±4.98%
events/ee-emit.js listeners=1 argc=10 n=2000000 *** 114.85 % ±5.35% ±7.20% ±9.56%
events/ee-emit.js listeners=1 argc=2 n=2000000 *** 114.53 % ±3.65% ±4.92% ±6.52%
events/ee-emit.js listeners=1 argc=4 n=2000000 *** 119.26 % ±1.67% ±2.23% ±2.93%
events/ee-emit.js listeners=10 argc=0 n=2000000 *** 9.36 % ±1.44% ±1.91% ±2.49%
events/ee-emit.js listeners=10 argc=10 n=2000000 *** 8.07 % ±1.19% ±1.59% ±2.07%
events/ee-emit.js listeners=10 argc=2 n=2000000 *** 9.76 % ±1.08% ±1.45% ±1.90%
events/ee-emit.js listeners=10 argc=4 n=2000000 *** 8.57 % ±1.16% ±1.55% ±2.01%
events/ee-emit.js listeners=5 argc=0 n=2000000 *** 14.15 % ±1.51% ±2.02% ±2.63%
events/ee-emit.js listeners=5 argc=10 n=2000000 *** 12.37 % ±1.34% ±1.78% ±2.32%
events/ee-emit.js listeners=5 argc=2 n=2000000 *** 13.79 % ±1.42% ±1.89% ±2.47%
events/ee-emit.js listeners=5 argc=4 n=2000000 *** 11.68 % ±3.91% ±5.25% ±6.93%
events/ee-listener-count-on-prototype.js n=50000000 *** -9.40 % ±1.30% ±1.75% ±2.32%
events/ee-listeners.js raw='false' listeners=5 n=5000000 *** -5.34 % ±2.32% ±3.12% ±4.12%
events/ee-listeners.js raw='false' listeners=50 n=5000000 * 1.49 % ±1.16% ±1.55% ±2.03%
events/ee-listeners.js raw='true' listeners=5 n=5000000 *** -7.73 % ±1.63% ±2.19% ±2.89%
events/ee-listeners.js raw='true' listeners=50 n=5000000 *** -5.43 % ±0.91% ±1.21% ±1.59%
events/ee-once.js argc=0 n=20000000 *** 16.98 % ±1.47% ±1.95% ±2.55%
events/ee-once.js argc=1 n=20000000 *** 16.58 % ±1.34% ±1.79% ±2.34%
events/ee-once.js argc=4 n=20000000 *** 15.86 % ±1.17% ±1.57% ±2.06%
events/ee-once.js argc=5 n=20000000 *** 15.28 % ±1.63% ±2.17% ±2.84%
events/eventtarget.js listeners=1 n=1000000 -0.15 % ±0.46% ±0.62% ±0.80%
events/eventtarget.js listeners=10 n=1000000 0.01 % ±0.50% ±0.66% ±0.87%
events/eventtarget.js listeners=5 n=1000000 -0.45 % ±0.54% ±0.73% ±0.95%
Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 28 comparisons, you can thus expect the following amount of false-positive results:
1.40 false positives, when considering a 5% risk acceptance (*, **, ***),
0.28 false positives, when considering a 1% risk acceptance (**, ***),
0.03 false positives, when considering a 0.1% risk acceptance (***)
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
theanarkh, abdo643-HULK, ardatan, bricss, MomenNano, ycjcl868, rinodrummer, shqld, hasegeli, Theofilos-Chamalis, and 2 more reacted with thumbs up emoji
ardatan, seahindeniz, metcoder95, bricss, MomenNano, tinchoz49, and tryb3l reacted with hooray emoji
bricss, distracteddev, MomenNano, egeominotti, yasarcelep, and tryb3l reacted with rocket emoji
tonivj5, ardatan, gurgunday, seahindeniz, bricss, MomenNano, and gamelaster reacted with eyes emoji
All reactions
-
👍
12 reactions
-
🎉
7 reactions
-
🚀
6 reactions
-
👀
7 reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
|
this is extremely interesting! well done on the find
|
tonivj5, StefanFeederle, MomenNano, and hood reacted with thumbs up emoji
anonrig, MomenNano, Theofilos-Chamalis, j0nimost, and LuXDAmore reacted with rocket emoji
All reactions
-
👍
4 reactions
-
🚀
5 reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
the previous PR seems to be stalled
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I think a key benchmark that should be added and checked here is one that uses a configurable number of unique event handlers to potentially highlight the difference between "fast mode" (best when very few properties added to it) and "slow mode" objects (best when lots of properties added to it), kinda like benchmark/es/map-bench.js but with key names that are not dependent upon the number of loop iterations.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Beware this causes a shared prototype { proto: null } instead of object.create doesn't and should be benchmarks as it does change behavior however slight
|
juliangruber reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
It should also be noted that we used to use this exact same storage method until #11930.
|
lpinca and aduh95 reacted with thumbs up emoji
tonivj5 and juliangruber reacted with eyes emoji
All reactions
-
👍
2 reactions
-
👀
2 reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
aduh95
commented
Sep 14, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1191/
Results
confidence improvement accuracy (*) (**) (***)
events/ee-add-remove.js n=1000000 removeListener=0 newListener=0 *** 11.10 % ±3.59% ±4.82% ±6.35%
events/ee-add-remove.js n=1000000 removeListener=0 newListener=1 *** 24.50 % ±3.28% ±4.37% ±5.72%
events/ee-add-remove.js n=1000000 removeListener=1 newListener=0 *** 8.60 % ±4.33% ±5.81% ±7.64%
events/ee-add-remove.js n=1000000 removeListener=1 newListener=1 3.22 % ±3.94% ±5.25% ±6.84%
events/ee-emit.js listeners=10 argc=0 n=2000000 *** 6.57 % ±1.54% ±2.05% ±2.66%
events/ee-emit.js listeners=10 argc=10 n=2000000 *** 8.54 % ±1.18% ±1.57% ±2.04%
events/ee-emit.js listeners=10 argc=2 n=2000000 *** 6.27 % ±1.37% ±1.82% ±2.37%
events/ee-emit.js listeners=10 argc=4 n=2000000 *** 9.21 % ±1.36% ±1.81% ±2.37%
events/ee-emit.js listeners=1 argc=0 n=2000000 *** 146.74 % ±7.88% ±10.54% ±13.83%
events/ee-emit.js listeners=1 argc=10 n=2000000 *** 141.56 % ±9.43% ±12.60% ±16.52%
events/ee-emit.js listeners=1 argc=2 n=2000000 *** 155.60 % ±9.92% ±13.31% ±17.55%
events/ee-emit.js listeners=1 argc=4 n=2000000 *** 166.25 % ±21.89% ±29.45% ±39.02%
events/ee-emit.js listeners=5 argc=0 n=2000000 *** 13.69 % ±3.19% ±4.24% ±5.52%
events/ee-emit.js listeners=5 argc=10 n=2000000 *** 6.82 % ±3.51% ±4.67% ±6.09%
events/ee-emit.js listeners=5 argc=2 n=2000000 *** 11.49 % ±3.55% ±4.74% ±6.20%
events/ee-emit.js listeners=5 argc=4 n=2000000 *** 6.43 % ±2.49% ±3.31% ±4.31%
events/ee-listener-count-on-prototype.js n=50000000 *** -7.27 % ±2.11% ±2.81% ±3.66%
events/ee-listeners.js raw='false' listeners=50 n=5000000 -0.75 % ±1.79% ±2.39% ±3.13%
events/ee-listeners.js raw='false' listeners=5 n=5000000 -1.80 % ±3.50% ±4.67% ±6.13%
events/ee-listeners.js raw='true' listeners=50 n=5000000 * -2.14 % ±2.00% ±2.67% ±3.48%
events/ee-listeners.js raw='true' listeners=5 n=5000000 ** -6.56 % ±3.94% ±5.24% ±6.83%
events/ee-once.js argc=0 n=20000000 *** 19.11 % ±2.15% ±2.88% ±3.78%
events/ee-once.js argc=1 n=20000000 *** 19.71 % ±2.23% ±2.97% ±3.88%
events/ee-once.js argc=4 n=20000000 *** 18.59 % ±1.28% ±1.71% ±2.23%
events/ee-once.js argc=5 n=20000000 *** 19.69 % ±1.70% ±2.27% ±2.97%
events/eventtarget.js listeners=10 n=1000000 0.74 % ±5.87% ±7.81% ±10.18%
events/eventtarget.js listeners=1 n=1000000 -4.58 % ±4.94% ±6.62% ±8.71%
events/eventtarget.js listeners=5 n=1000000 -1.30 % ±2.18% ±2.90% ±3.79%
Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 28 comparisons, you can thus
expect the following amount of false-positive results:
1.40 false positives, when considering a 5% risk acceptance (*, **, ***),
0.28 false positives, when considering a 1% risk acceptance (**, ***),
0.03 false positives, when considering a 0.1% risk acceptance (***)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
mscdex
commented
Sep 14, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
I created a benchmark as described in #44627 (comment) and the results are basically what I suspected:
confidence improvement accuracy (*) (**) (***)
events/ee-add-remove-multiple.js n=1000000 events=1 *** 79.91 % ±2.11% ±2.83% ±3.74%
events/ee-add-remove-multiple.js n=1000000 events=2 *** -14.78 % ±0.88% ±1.18% ±1.54%
events/ee-add-remove-multiple.js n=1000000 events=3 *** -10.03 % ±1.25% ±1.66% ±2.17%
events/ee-add-remove-multiple.js n=1000000 events=5 *** -7.24 % ±1.12% ±1.49% ±1.95%
Code
'use strict';
const common = require('../common.js');
const { EventEmitter } = require('events');
const bench = common.createBenchmark(main, {
events: [1, 2, 3, 5],
n: [1e6],
});
function main({ events, n }) {
const ee = new EventEmitter();
const listeners = [];
for (let k = 0; k < 10; k += 1)
listeners.push(() => {});
const eventNames = [];
for (let k = 0; k < events; ++k)
eventNames.push(`dummy${k}`);
bench.start();
for (let i = 0; i < n; i += 1) {
for (const eventName of eventNames) {
for (let k = listeners.length; --k >= 0; /* empty */) {
ee.on(eventName, listeners[k]);
}
}
for (const eventName of eventNames) {
for (let k = listeners.length; --k >= 0; /* empty */) {
ee.removeListener(eventName, listeners[k]);
}
}
}
bench.end(n);
}
In this case it only takes two unique events for the regression to show up, which is a bit surprising as I thought it used to take more for V8 to transition objects to slow mode. After seeing these results, I don't think the changes in this PR are worth it, considering most event emitters will see handlers added for multiple events and not just one.
|
Gordapanela reacted with confused emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
anonrig
commented
Sep 15, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
@mscdex I basically run the same benchmark on my machine (M1 Max, 64 GB, arm64), and added more values to events (10 and 20), and my results are a lot different than yours:
confidence improvement accuracy (*) (**) (***)
events/ee-listen-unique.js n=1000000 events=1 *** 75.80 % ±1.42% ±1.89% ±2.46%
events/ee-listen-unique.js n=1000000 events=10 * -2.07 % ±1.83% ±2.43% ±3.17%
events/ee-listen-unique.js n=1000000 events=2 *** -4.48 % ±1.03% ±1.37% ±1.80%
events/ee-listen-unique.js n=1000000 events=20 0.56 % ±3.92% ±5.22% ±6.79%
events/ee-listen-unique.js n=1000000 events=3 * -1.91 % ±1.55% ±2.06% ±2.69%
events/ee-listen-unique.js n=1000000 events=5 *** -3.86 % ±1.73% ±2.30% ±2.99%
Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 6 comparisons, you can thus expect the following amount of false-positive results:
0.30 false positives, when considering a 5% risk acceptance (*, **, ***),
0.06 false positives, when considering a 1% risk acceptance (**, ***),
0.01 false positives, when considering a 0.1% risk acceptance (***)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
mscdex
commented
Sep 15, 2022
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
I ran my benchmarks on an i7-3770K running Linux 5.4.0 (and compared using benchmark/compare.R if that makes a difference) 🤷
I also ran with larger values for events but the trend is pretty clear anyway.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
I also ran with larger values for events but the trend is pretty clear anyway.
I agree. I will continue working on this. Can you share with me more information about the fast/slow objects in the v8? That would be educational for everybody.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
v8 tries to optimize access to objects based on their "shape". If the shape changes too much, v8 gives up trying to optimize it and switches into a fallback "dictionary" mode, which is basically just a raw hash map. The properties of an object and the types of the values make up the shape (to some degree, check out https://mathiasbynens.be/notes/shapes-ics for an in depth explanation), so adding new properties to an object (let x = {}; x.a = 1;) or deleting properties (let x = {a: 1}; delete x.a;), among other things, will eventually make v8 give up.
in bad hacky idea land, we could manually try calling %ToFastProperties on eventemitters to make them "recover" from the slow state, though it would be difficult to know when it was a good time to call it.
|
anonrig, mcollina, styfle, and tryb3l reacted with heart emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
In any case, we may want to land the benchmark, that seems a good idea to have one to catch those kind of perf regression in the future.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should not land due to regressions.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
|
if (this._events === undefined ||
|
|
|
this._events === ObjectGetPrototypeOf(this)._events) {
|
|
|
this._events = ObjectCreate(null);
|
|
|
this._events = new NullObject();
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is fairly problematic, I think. At the very least it's semver-major. Unfortunately we have a number of ecosystem modules that touch this._events directly and changing the type of thing that is can definitely break stuff.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I don't want to be rude. I would be grateful for any input from EventEmitter experts here.
I updated the benchmark script attached in this thread to determine if it is reasonable to publish and subscribe thousands of different events on an EventEmitter instance (of course there is a warning when more than 10 event ids are attached).
It looks like this is a terrible idea, but if anyone here would confirm that it is terrible, it would close the case for me once and for all. Thank you for any reply.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
jasnell
jasnell left review comments
JungMinu
JungMinu approved these changes
+1 more reviewer
Reviewers whose approvals may not affect merge requirements
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Labels
events
Issues and PRs related to EventEmitter and the events module.
needs-ci
PRs that need a full CI run.
performance
Issues and PRs related to the performance of Node.js.
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|