GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — perf_hooks: add statistical hypothesis testing to histogram · nodejs/node@b9d7a50

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

doc/api
- perf_hooks.md
lib/internal
- histogram.js
src
test/parallel
- test-perf-hooks-histogram-stats.js

Expand file treeCollapse file tree

Open diff view settings

Collapse file

`‎doc/api/perf_hooks.md‎`

Copy file name to clipboardExpand all lines: doc/api/perf_hooks.md

+255Lines changed: 255 additions & 0 deletions

Display the source diff
Display the rich diff

Original file line number	Diff line number	Diff line change
`@@ -1621,6 +1621,16 @@ added:`
`1621`	`1621`	Default: `Number.MAX_SAFE_INTEGER`.
`1622`	`1622`	* `figures` {number} The number of accuracy digits. Must be a number between
`1623`	`1623`	`1` and `5`. Default: `3`.
	`1624`	+ * `halfLife` {number} The EWMA half-life in number of samples. When set to
	`1625`	`+ a value greater than 0, the histogram tracks an exponentially weighted`
	`1626`	`+ moving average and standard deviation, accessible via`
	`1627`	+ `histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
	`1628`	+ recordings, a value's influence has decayed to 50%. Default: `0`
	`1629`	`+ (disabled).`
	`1630`	+ * `threshold` {number} An SLO threshold value. When set together with
	`1631`	+ `halfLife`, the histogram tracks a smoothed error rate for values
	`1632`	+ exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
	`1633`	+ `histogram.burnRate()`. Default: `0` (disabled).
`1624`	`1634`	`* Returns: {RecordableHistogram}`
`1625`	`1635`
`1626`	`1636`	`Returns a {RecordableHistogram}.`
`@@ -1885,6 +1895,36 @@ value, representing the probability that a recorded value will be less`
`1885`	`1895`	than or equal to `value`. This is the inverse operation of
`1886`	`1896`	`histogram.percentile()`.
`1887`	`1897`
	`1898`	+### `histogram.cliffsD(other)`
	`1899`	`+`
	`1900`	`+<!-- YAML`
	`1901`	`+added: REPLACEME`
	`1902`	`+-->`
	`1903`	`+`
	`1904`	+* `other` {Histogram} The histogram to compare against.
	`1905`	`+* Returns: {number} A value between -1.0 and 1.0.`
	`1906`	`+`
	`1907`	`+Computes [Cliff's delta][], a non-parametric effect size measure. Returns`
	`1908`	`+the probability that a random value from this histogram exceeds a random`
	`1909`	+value from `other`, minus the reverse probability. A value of 1 means every
	`1910`	+value in this histogram exceeds every value in `other`; -1 means the
	`1911`	`+opposite; 0 means no tendency in either direction.`
	`1912`	`+`
	`1913`	+### `histogram.cohensD(other)`
	`1914`	`+`
	`1915`	`+<!-- YAML`
	`1916`	`+added: REPLACEME`
	`1917`	`+-->`
	`1918`	`+`
	`1919`	+* `other` {Histogram} The histogram to compare against.
	`1920`	`+* Returns: {number} The effect size.`
	`1921`	`+`
	`1922`	`+Computes [Cohen's d][] effect size, the standardized difference between the`
	`1923`	+means of this histogram and `other`, using the pooled standard deviation.
	`1924`	`+Positive values indicate this histogram has a higher mean. By convention,`
	`1925`	`+\|d\| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.`
	`1926`	`+Both histograms must have at least 2 recorded values; otherwise returns 0.`
	`1927`	`+`
`1888`	`1928`	### `histogram.countAt(value)`
`1889`	`1929`
`1890`	`1930`	`<!-- YAML`
`@@ -1921,6 +1961,74 @@ added:`
`1921`	`1961`	`The number of times the event loop delay exceeded the maximum 1 hour event`
`1922`	`1962`	`loop delay threshold.`
`1923`	`1963`
	`1964`	+### `histogram.ewmaMean`
	`1965`	`+`
	`1966`	`+<!-- YAML`
	`1967`	`+added: REPLACEME`
	`1968`	`+-->`
	`1969`	`+`
	`1970`	`+* Type: {number}`
	`1971`	`+`
	`1972`	`+The exponentially weighted moving average of recorded values. Only active`
	`1973`	+when the histogram was created with a `halfLife` option greater than 0.
	`1974`	+Returns `0` when EWMA is disabled or no values have been recorded.
	`1975`	`+`
	`1976`	+### `histogram.ewmaStddev`
	`1977`	`+`
	`1978`	`+<!-- YAML`
	`1979`	`+added: REPLACEME`
	`1980`	`+-->`
	`1981`	`+`
	`1982`	`+* Type: {number}`
	`1983`	`+`
	`1984`	`+The exponentially weighted moving standard deviation. Only active when the`
	`1985`	+histogram was created with a `halfLife` option greater than 0. Returns `0`
	`1986`	`+when EWMA is disabled or no values have been recorded.`
	`1987`	`+`
	`1988`	+### `histogram.ewmaErrorRate`
	`1989`	`+`
	`1990`	`+<!-- YAML`
	`1991`	`+added: REPLACEME`
	`1992`	`+-->`
	`1993`	`+`
	`1994`	`+* Type: {number}`
	`1995`	`+`
	`1996`	`+The EWMA-smoothed probability of a recorded value exceeding the configured`
	`1997`	+`threshold`. Only active when the histogram was created with both `halfLife`
	`1998`	+and `threshold` options. Returns `0` when not enabled or no values have been
	`1999`	`+recorded.`
	`2000`	`+`
	`2001`	+### `histogram.burnRate(sloTarget)`
	`2002`	`+`
	`2003`	`+<!-- YAML`
	`2004`	`+added: REPLACEME`
	`2005`	`+-->`
	`2006`	`+`
	`2007`	+* `sloTarget` {number} The SLO target as a fraction between 0 and 1
	`2008`	+ (exclusive). For example, `0.999` for a 99.9% SLO.
	`2009`	`+* Returns: {number}`
	`2010`	`+`
	`2011`	+Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
	`2012`	`+of 1 means the error budget will be exactly exhausted over the SLO window.`
	`2013`	`+A burn rate greater than 1 means it is being consumed faster than allowed.`
	`2014`	+Requires the histogram to have been created with both `halfLife` and
	`2015`	+`threshold` options.
	`2016`	`+`
	`2017`	+```js
	`2018`	`+const { createHistogram } = require('node:perf_hooks');`
	`2019`	`+`
	`2020`	`+// Track latency with a 200ms SLO threshold, half-life of 100 samples`
	`2021`	`+const h = createHistogram({ halfLife: 100, threshold: 200_000_000 });`
	`2022`	`+`
	`2023`	`+// ... record latency values ...`
	`2024`	`+`
	`2025`	`+// Check burn rate against a 99.9% SLO`
	`2026`	`+const rate = h.burnRate(0.999);`
	`2027`	`+if (rate > 1) {`
	`2028`	+ console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
	`2029`	`+}`
	`2030`	+```
	`2031`	`+`
`1924`	`2032`	### `histogram.ksTest(other)`
`1925`	`2033`
`1926`	`2034`	`<!-- YAML`
`@@ -1974,6 +2082,24 @@ Returns the histogram data rebucketed into logarithmically-spaced`
`1974`	`2082`	intervals, where each bucket's width is multiplied by `base`.
`1975`	`2083`	`Useful for visualization and export.`
`1976`	`2084`
	`2085`	+### `histogram.mannWhitneyTest(other)`
	`2086`	`+`
	`2087`	`+<!-- YAML`
	`2088`	`+added: REPLACEME`
	`2089`	`+-->`
	`2090`	`+`
	`2091`	+* `other` {Histogram} The histogram to compare against.
	`2092`	`+* Returns: {Object}`
	`2093`	+ * `uStatistic` {number} The Mann-Whitney U statistic.
	`2094`	+ * `zScore` {number} The z-score (normal approximation).
	`2095`	+ * `pValue` {number} Two-tailed p-value.
	`2096`	`+`
	`2097`	`+Performs a [Mann-Whitney U test][] comparing whether this histogram tends to`
	`2098`	+produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
	`2099`	`+non-parametric test that makes no assumptions about the shape of the`
	`2100`	`+distributions. Uses the normal approximation with tie correction for the`
	`2101`	`+p-value.`
	`2102`	`+`
`1977`	`2103`	### `histogram.max`
`1978`	`2104`
`1979`	`2105`	`<!-- YAML`
`@@ -2052,6 +2178,40 @@ added:`
`2052`	`2178`
`2053`	`2179`	`Returns the value at the given percentile.`
`2054`	`2180`
	`2181`	+### `histogram.percentileCI(percentile[, options])`
	`2182`	`+`
	`2183`	`+<!-- YAML`
	`2184`	`+added: REPLACEME`
	`2185`	`+-->`
	`2186`	`+`
	`2187`	+* `percentile` {number} A percentile value in the range (0, 100].
	`2188`	+* `options` {Object}
	`2189`	+ * `confidence` {number} The confidence level for the interval, between
	`2190`	+ 0 and 1 (exclusive). Default: `0.95`.
	`2191`	`+* Returns: {Object}`
	`2192`	+ * `value` {number} The point estimate (same as `histogram.percentile()`).
	`2193`	+ * `lower` {number} The lower bound of the confidence interval.
	`2194`	+ * `upper` {number} The upper bound of the confidence interval.
	`2195`	`+`
	`2196`	`+Returns a confidence interval for the given percentile using the exact`
	`2197`	`+binomial method. With fewer samples, the interval will be wider, reflecting`
	`2198`	`+the greater uncertainty in the percentile estimate. Requires at least 2`
	`2199`	+recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
	`2200`	`+`
	`2201`	+```js
	`2202`	`+const { createHistogram } = require('node:perf_hooks');`
	`2203`	`+`
	`2204`	`+const h = createHistogram();`
	`2205`	`+for (let i = 0; i < 1000; i++) {`
	`2206`	`+ h.record(Math.floor(Math.random() * 100));`
	`2207`	`+}`
	`2208`	`+`
	`2209`	`+const ci = h.percentileCI(99);`
	`2210`	`+console.log(ci.value); // The p99 point estimate`
	`2211`	`+console.log(ci.lower); // The lower bound (95% confidence)`
	`2212`	`+console.log(ci.upper); // The upper bound (95% confidence)`
	`2213`	+```
	`2214`	`+`
`2055`	`2215`	### `histogram.percentiles`
`2056`	`2216`
`2057`	`2217`	`<!-- YAML`
`@@ -2119,6 +2279,31 @@ added: v11.10.0`
`2119`	`2279`
`2120`	`2280`	`The standard deviation of the recorded event loop delays.`
`2121`	`2281`
	`2282`	+### `histogram.welchTest(other[, options])`
	`2283`	`+`
	`2284`	`+<!-- YAML`
	`2285`	`+added: REPLACEME`
	`2286`	`+-->`
	`2287`	`+`
	`2288`	+* `other` {Histogram} The histogram to compare against.
	`2289`	+* `options` {Object}
	`2290`	+ * `confidence` {number} Confidence level for the interval, between 0 and 1.
	`2291`	+ Default: `0.95`.
	`2292`	`+* Returns: {Object}`
	`2293`	+ * `tStatistic` {number} The Welch t-statistic.
	`2294`	+ * `degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
	`2295`	+ * `pValue` {number} Two-tailed p-value.
	`2296`	+ * `confidenceInterval` {Object}
	`2297`	+ * `lower` {number} Lower bound of the confidence interval on the
	`2298`	`+ difference of means.`
	`2299`	+ * `upper` {number} Upper bound.
	`2300`	`+`
	`2301`	+Performs [Welch's t-test][] comparing the means of this histogram and `other`.
	`2302`	`+The p-value indicates the probability of observing a difference at least this`
	`2303`	`+extreme under the null hypothesis that the two distributions have the same`
	`2304`	`+mean. Both histograms must have at least 2 recorded values; otherwise the`
	`2305`	+result has `pValue` 1 and `tStatistic` 0.
	`2306`	`+`
`2122`	`2307`	## Class: `ELDHistogram extends Histogram`
`2123`	`2308`
`2124`	`2309`	A `Histogram` that records event loop delay, returned by
`@@ -2280,6 +2465,32 @@ const violating = latency.ccdf(500_000_000);`
`2280`	`2465`	console.log(`${(violating * 100).toFixed(1)}% of requests violating SLO`);
`2281`	`2466`	```
`2282`	`2467`
	`2468`	`+### SLO burn rate monitoring`
	`2469`	`+`
	`2470`	+```js
	`2471`	`+const { createHistogram } = require('node:perf_hooks');`
	`2472`	`+`
	`2473`	`+// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold`
	`2474`	`+const latency = createHistogram({`
	`2475`	`+ halfLife: 100,`
	`2476`	`+ threshold: 200_000_000, // 200ms in nanoseconds`
	`2477`	`+});`
	`2478`	`+`
	`2479`	`+// Record request latencies...`
	`2480`	`+`
	`2481`	`+// Smoothed error rate: probability of exceeding the threshold`
	`2482`	+console.log(`Error rate: ${(latency.ewmaErrorRate * 100).toFixed(2)}%`);
	`2483`	`+`
	`2484`	`+// Burn rate against a 99.9% SLO`
	`2485`	`+// >1 means the error budget is depleting faster than allowed`
	`2486`	`+const rate = latency.burnRate(0.999);`
	`2487`	+console.log(`Burn rate: ${rate.toFixed(2)}x`);
	`2488`	`+`
	`2489`	`+// EWMA mean and stddev track the smoothed latency`
	`2490`	+console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
	`2491`	+console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
	`2492`	+```
	`2493`	`+`
`2283`	`2494`	`### Regression detection with KS test`
`2284`	`2495`
`2285`	`2496`	```js
`@@ -2331,6 +2542,46 @@ newSnapshot.subtract(snapshot);`
`2331`	`2542`	`console.log('Recent p99:', newSnapshot.percentile(99));`
`2332`	`2543`	```
`2333`	`2544`
	`2545`	`+### Benchmark comparison with Welch's t-test`
	`2546`	`+`
	`2547`	+```js
	`2548`	`+const { createHistogram } = require('node:perf_hooks');`
	`2549`	`+`
	`2550`	`+const baseline = createHistogram();`
	`2551`	`+const candidate = createHistogram();`
	`2552`	`+`
	`2553`	`+// Record operation rates from the old and new builds...`
	`2554`	`+`
	`2555`	`+const result = baseline.welchTest(candidate);`
	`2556`	`+const improvement = ((candidate.mean - baseline.mean) / baseline.mean * 100);`
	`2557`	`+`
	`2558`	+console.log(`Improvement: ${improvement.toFixed(2)}%`);
	`2559`	+console.log(`p-value: ${result.pValue.toFixed(6)}`);
	`2560`	+console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, ` +
	`2561`	+ `${result.confidenceInterval.upper.toFixed(2)}]`);
	`2562`	`+`
	`2563`	`+if (result.pValue < 0.05) {`
	`2564`	`+ const d = baseline.cohensD(candidate);`
	`2565`	+ console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
	`2566`	`+}`
	`2567`	+```
	`2568`	`+`
	`2569`	`+### Effect size with Cliff's delta`
	`2570`	`+`
	`2571`	+```js
	`2572`	`+const { createHistogram } = require('node:perf_hooks');`
	`2573`	`+`
	`2574`	`+const before = createHistogram();`
	`2575`	`+const after = createHistogram();`
	`2576`	`+`
	`2577`	`+// Record latencies before and after a change...`
	`2578`	`+`
	`2579`	`+const delta = before.cliffsD(after);`
	`2580`	`+// A delta > 0: before tends to produce larger values (improvement)`
	`2581`	`+// A delta < 0: after tends to produce larger values (regression)`
	`2582`	+console.log(`Cliff's delta: ${delta.toFixed(4)}`);
	`2583`	+```
	`2584`	`+`
`2334`	`2585`	`## Examples`
`2335`	`2586`
`2336`	`2587`	`### Measuring the duration of async operations`
`@@ -2585,13 +2836,17 @@ dns.promises.resolve('localhost');`
`2585`	`2836`	```
`2586`	`2837`
`2587`	`2838`	`[Async Hooks]: async_hooks.md`
	`2839`	`+[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta`
	`2840`	`+[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d`
`2588`	`2841`	`[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info`
`2589`	`2842`	`[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info`
`2590`	`2843`	`[High Resolution Time]: https://www.w3.org/TR/hr-time-2`
	`2844`	`+[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test`
`2591`	`2845`	`[Performance Timeline]: https://w3c.github.io/performance-timeline/`
`2592`	`2846`	`[Resource Timing]: https://www.w3.org/TR/resource-timing-2/`
`2593`	`2847`	`[User Timing]: https://www.w3.org/TR/user-timing/`
`2594`	`2848`	`[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/`
	`2849`	`+[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test`
`2595`	`2850`	`[Worker threads]: worker_threads.md#worker-threads`
`2596`	`2851`	[`'exit'`]: process.md#event-exit
`2597`	`2852`	[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options

0 commit comments

Comments

0 (0)

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.