Labs is JS benchmarking you can trust, with CPU and GPU benchmarks across multiple runtimes. Trying to get good signal is harder than you might think. VMs are non-deterministic, environments are unstable, and typical benchmarks don't give you any sense of a comparison's validity. Labs detects variance, giving feedback on how to fix it, and uses statistical analysis to determine if two runs are actually different.

npm i @pmndrs/labs

Create a config and a bench file. Benches use a generator where code before yield is setup, the yielded function is measured, and code after is teardown.

// labs.config.ts
import { defineConfig } from '@pmndrs/labs'

export default defineConfig({
  benchDir: '.',
})

Use @tags in the name string for filtering.

// array-push.bench.ts
import { bench, group } from '@pmndrs/labs'

group('array @stress', () => {
  bench('push 1k', function* () {
    const arr: number[] = []
    yield () => {
      for (let i = 0; i < 1000; i++) arr.push(i)
    }
  })
})

Run your benchmarks with the bench command.

# Run all benches, save named after the current commit
bench
# Or choose another runtime
bench --runtime chrome # bun | webkit | firefox
# Or filter by file name and tag
bench vec3 matrix @mytag
# Save with a name for easier reading
bench "@mytag" -n 'v1.0.0'

In an interactive terminal, a live view shows each benchmark's progress while it runs, then a full-screen results view opens. Select a benchmark to inspect its timings, sample distribution, memory use, and run consistency. Quitting prints a short summary. Reopen saved results with bench list.

Compare against a baseline.

bench compare

Browse the results in a full-screen comparison view. Select a benchmark to see its timings, memory use, run consistency, and confidence interval.

Alternatively, compare the working tree against a git revision or another checkout. Labs alternates fresh process blocks between both trees in one session to reduce differences in clock speed, thermals, and background load. Both runs are saved, and the reference is compared against the candidate.

bench --against main
bench --against ../wt-a --candidate ../wt-b

Use bench.gpu to time WebGPU compute or render passes. The generator receives a device for setup and yields the pass to measure.

import { bench } from '@pmndrs/labs'

bench.gpu('blur', function* ({ device }) {
  const { pipeline, bind } = build(device)
  yield {
    compute: (pass) => {
      pass.setPipeline(pipeline)
      pass.setBindGroup(0, bind)
      pass.dispatchWorkgroups(4096)
    },
  }
})

Run GPU benches in Chrome or WebKit. CPU and GPU benches can share a file and interleave their blocks.

bench --runtime chrome # or webkit

Results show time per pass and VRAM use. See the GPU example for a complete suite.

Labs promises to give results you can trust. To do this a number of guarantees are made when running benches.

• Each bench block runs in its own isolated worker process, preventing benches in the same file from contaminating each other's JIT state, heap layout, or GC history. Reordering benches can otherwise skew results by 2x or more.
• Each block warms up before timing. Observed optimization changes are reported, and deoptimization triggers a warning without discarding samples.
• Samples run consecutively after one initial GC. Natural GC remains in timing, all samples are retained, and heap diagnostics run separately afterward.
• Every bench is run in isolated blocks that are interleaved: A₁ B₁ C₁ → A₂ B₂ C₂ → … → A₈ B₈ C₈. This reduces bias from gradual changes such as CPU throttling or even boosting.
• Timing overhead is controlled. If a sample's measurement time is so fast that the overhead of the timing itself would bias the results, then it is run in a batch sized so each sample lasts about a millisecond.
• Detect dead code elimination (DCE). If the samples measure the same as an empty function call, then we detect DCE and report it. This can be mitigated by returning a result from the yielded function.
• Detect non-deterministic benches. When warmup timings never settle or the output differs between fresh processes, a warning is given.
• Comparison precision is measured. When median timings vary too much across blocks to detect a minDelta change, a warning is given with advice: rerun on an idle machine when the CPU speed probes explain the variation, otherwise add blocks.

A faster result only matters if the work is still correct. Labs keeps the result of the first untimed call and hands it back from yield once measurement finishes, so teardown can check it. Use any assertion library, or the small assert included with Labs. A failed check exits with code 1, keeps the saved result out of the baseline, and stops the remaining blocks of that bench.

import { assert, bench } from '@pmndrs/labs'

bench('sort', function* () {
  const result = yield () => [3, 1, 2].toSorted()
  assert.equal(result, [1, 2, 3])
})

Return the result from the generator to compare it automatically against the baseline's output.

bench('sort', function* () {
  const result = yield () => [3, 1, 2].toSorted()
  return result
})

For work that mutates state, use a snapshot hook to select the output.

bench('append', function* () {
  const values: number[] = []
  yield {
    bench: () => values.push(values.length),
    snapshot: () => values,
  }
})

One of the largest sources of noise when running benchmarks is an unstable environment, and the usual culprit is the CPU. The CPU boosts or thermal throttles, or a process gets put on a P-core (performance) instead of an E-core (efficiency). Labs checks the CPU clocks before and after each benchmark file and before each block, and tracks whether they vary across the runs. If it detects too much variance, you will get warned and the run will be flagged. But what can you do about it?

You get decent control with Windows by going into the BIOS. 90% of the variance is solved by disabling any kind of CPU turbo.

• Boot into the BIOS and disable turbo + SMT.
• Run the benchmarks on the highest priority.
• Disable as much background tasks as you can.

While Apple Silicon is relatively stable, there isn't much that can be done to control it. The governor cannot be adjusted and the dynamic CPU frequencies cannot be disabled. Expect occasional unstable clock warnings. They do not always mean the results are affected, so check whether run medians agree before discarding a run.

Linux gives the most controls getting the best possible environment for testing. See this LLVM guide for specifics.

Every run saves results by default, named after the current commit (abc1234, with -dirty when the tree has uncommitted changes, and a counter for repeat runs: abc1234-2). Outside a git repo, names fall back to a timestamp. Each result also records the commit, branch, and dirty state it was produced from. bench run is an alias for bench; pass --no-save to execute without saving.

pnpm bench                              # run all, save named after the current commit
pnpm bench "relation"                   # partial match on file name, save
pnpm bench "relation churn"             # separator-agnostic match, save
pnpm bench "@relation"                  # filter by tag, save
pnpm bench churn @relation              # name + tag combined, save
pnpm bench --help                       # show usage
pnpm bench -n "v1.2.0"                 # save with explicit name (prompts if it exists)
pnpm bench -n "v1.2.0" -f              # overwrite existing without prompting
pnpm bench -n "v1.2.0" --force         # same as -f
pnpm bench -n "v1.2.0" -m "refactor"   # save with name and description
pnpm bench --baseline                   # save and set as baseline
pnpm bench -b                           # shorthand for --baseline
pnpm bench -n "v1.2.0" -b              # save with name and set as baseline
pnpm bench --compare                    # save, then compare vs baseline
pnpm bench --blocks 12                  # save with 12 fresh-process blocks per benchmark
pnpm bench --against main               # run the working tree and main interleaved, save both, compare
pnpm bench --against ../a --candidate ../b  # run two existing checkouts interleaved and compare
pnpm bench -c                           # shorthand for --compare

Results are saved to <benchDir>/.labs/results/<name>.json and include hardware metadata (CPU, arch, runtime) for like-for-like comparisons.

pnpm bench --no-save                    # run all, no save
pnpm bench run --no-save "relation"    # filtered, no save
pnpm bench --no-save "@relation"       # filtered by tag, no save
pnpm bench --no-save --blocks 3        # no save, with 3 blocks for faster feedback

pnpm bench list                        # list all saved results
pnpm bench delete "v1.2.0"             # delete a specific saved result
pnpm bench prune                       # remove results with unstable CPU clocks
pnpm bench clear                       # delete all saved results

bench list shows each result's name, description, timestamp, and CPU. The current baseline is marked with (baseline).

pnpm bench baseline                    # interactive baseline picker
pnpm bench baseline "v1.2.0"          # set a result as the baseline
pnpm bench --baseline                 # save and set the new result as baseline
pnpm bench -b                         # shorthand for --baseline

pnpm bench compare                     # interactive picker (latest preselected)
pnpm bench compare "v1.3.0"           # compare named result vs baseline
pnpm bench compare "v1.2.0" "v1.3.0"  # compare two saved results, baseline first
pnpm bench compare --last             # replay the last compared pair
pnpm bench compare -l                 # shorthand for --last

--against <rev|dir> and --candidate <rev|dir> each name a tree. An existing directory is used as it is, so worktrees other tools created can be compared directly, uncommitted changes included. Anything else is a git revision, checked out detached into a temporary worktree that is removed when the run ends. A tree without node_modules links the working checkout's for the run, so both sides resolve the same dependencies and only the code under test differs. Bench files are discovered in the working tree and matched by path in each side. A file one side lacks runs on the other side only. -n names the candidate's result, the reference's result is named after its commit, and the baseline pointer is neither read nor set. --baseline and --compare are rejected alongside --against. --candidate alone measures that tree in place of the working checkout.

In an interactive terminal, comparison opens a full-screen view with a scrolling benchmark list and a fixed inspector below it. Use ↑ / ↓, Page Up / Page Down, or Home / End to select a result. Press q or Esc to return to the previous terminal screen. Ctrl+C exits with code 130. The layout follows terminal resizing, and [ / ] scroll inspector details when the screen is short or a result has long warnings.

The inspector shows baseline and candidate medians, pooled-sample histograms on a shared scale, p99, GC, and heap per iteration. A shaded row shows the percentage changes. The p badge sits beside the benchmark name, and the footer summarizes faster, slower, and neutral results alongside the controls. Below the measurements:

• consistency ✦ N runs plots the actual fresh-process block medians in two adjacent rows, baseline in cyan above candidate in magenta. Single runs use regular dots, and multiple runs at the same plotted position use larger dots. Endpoint labels describe the shared time scale.
• spread ✦ 95% ci shows the Hodges-Lehmann effect and its confidence interval, with the bounds below the end caps. The shaded band is ±minDelta, and the central mark is zero. The confidence level follows alpha, and all benchmarks share the same effect scale.

Warnings remain available in the inspector. GC, heap, and p99 changes are descriptive, not independent statistical verdicts. NO_COLOR removes color while retaining the layout and verdict symbols.

Redirected output, CI, and TERM=dumb use the printable report. Its table starts at 96 columns with 28 columns for benchmark titles, and grows to fit longer titles or wider metric values:

Each row is prefixed with a verdict icon: green ▲ (faster), red ▼ (slower), or gray ■ (neutral). A red ✗ marks a bench whose snapshot differs from the baseline and shows output changed in place of a verdict. Candidate runs that failed a check or threw are listed under failed and receive no verdict. The verdict uses the Mann-Whitney p-value and the Hodges-Lehmann relative effect, not Δp50. In the printable report, below each row, two distribution sparklines sit under their respective columns — baseline (cyan) and candidate (magenta) — on a shared axis. The sparklines use pooled inner samples and are descriptive only. When both runs recorded gc or heap, the sparkline row ends with gc(…) and heap(…): the candidate's median value and its percent change from the baseline's median. The tags keep the same positions across rows, including when one metric is absent. These metrics carry no verdict and show — when the baseline is zero.

Comparison is gated. Two runs must pass environment checks before results are shown.

Environment checks (fail = entire comparison is denied):

• Hardware match — CPU model, architecture, and runtime (Node/Bun/etc.) must be identical between runs.

Environment warnings (non-blocking, printed above results):

• Clock drift — if either run's CPU frequency drifted > 5% during the run, a warning is shown. On Apple Silicon this is expected (no governor or turbo control); on other platforms it usually means turbo boost or thermal throttling is active.
• Clock speed mismatch — if the two runs' median clock speeds differ by > 5%, a warning is shown. Absolute timings may not be directly comparable.
• Block count mismatch — unequal counts are supported, but the actual counts determine whether the test can reach the configured significance level.

Per-benchmark eligibility and annotations:

• Not missing — the bench must exist in both runs. Benches present only in baseline or only in candidate are reported separately.
• Limited-resolution benches are annotated, not skipped — a bench whose approximate between-block resolution is coarser than minDelta still gets judged because the rank test already responds to spread. Its row carries a ⚠ ~±N% marker; a neutral result there is inconclusive at the shown resolution, not evidence of no change.
• Block replication — both sides need at least two fresh-process blocks, and their combined counts must permit an exact p-value at or below alpha. Results saved before blocked sampling are skipped.

Without an interactive terminal, labs never prompts. Commands that would open a picker or confirm an overwrite fail with the command to run instead, and bench list prints the saved names.

pnpm bench --json                       # judged results as JSON on stdout
pnpm bench -c --json --fail-on-slower   # also compare against the baseline
pnpm bench compare "v1.3.0" --json      # judged comparison of a saved result

With --json, stdout holds only the JSON document and everything else goes to stderr. Each bench has a status (passed, warning, or failed), a problem and advice when it has one, and its minDetectableChange. Comparison verdicts are faster, slower, neutral, inconclusive (neutral, but a minDelta change could have been missed), changed, failed, skipped, or missing.

Failed checks and changed output exit with 1. With --fail-on-slower, a slower bench also exits with 1, and a comparison that could have missed a change exits with 2: an inconclusive or skipped bench, or a candidate that ran on an unstable machine.

import { bench, group } from 'labs'

group('my-group @mytag', () => {
  bench('my-bench', function* () {
    // setup
    yield () => {
      const result = /* measured computation */ 1 + 1
      return result
    }
    // teardown
  })
})

bench.gpu times a pass on the GPU with timestamp queries. The generator receives the device and yields the pass. CPU work that uses the GPU, such as encoding or compiling, stays an ordinary bench that awaits gpu from its state. It never waits for the GPU, so it should encode and return the command buffer rather than submit it, or a compiled loop floods the queue.

import { bench } from '@pmndrs/labs'

bench.gpu('blur dispatch', function* ({ device }) {
  const { pipeline, bind } = build(device)
  yield {
    compute: (pass) => {
      pass.setPipeline(pipeline)
      pass.setBindGroup(0, bind)
      pass.dispatchWorkgroups(4096)
    },
  }
})

bench('blur compile', async function* ({ gpu, fresh }) {
  const device = await gpu
  yield async () => {
    const module = device.createShaderModule({ code: fresh(blurSource) })
    await device.createComputePipelineAsync({ layout: 'auto', compute: { module } })
  }
})

Render passes take their attachments: yield { render: descriptor, draw: (pass) => ... }. Work labs cannot wrap, such as WebGL timer queries, yields its own nanoseconds: yield { time: async () => ns }. fresh appends a unique comment so the driver's pipeline cache cannot serve a compile. See examples/gpu.

GPU benches run on chrome and webkit, which expose the machine's GPU headlessly. Headless Firefox has no WebGPU adapter. The GPU clock climbs under load and falls when idle, so passes warm up until their timings agree, and a probe dispatch records the GPU's speed per block for the clock gate. GPU rows read per pass and show vram, the buffers and textures held after setup with the net growth per pass, in place of GC and heap. Shader programs have no size the API exposes. The report adds a GPU clock line, and JSON carries clock per bench and gpuSpeedVaried.

Engines may remove pure computations whose results are never used, making a benchmark appear impossibly fast. When benchmarking pure work, return its result; Labs consumes non-undefined returns from automatically timed function benchmarks inside the timed region. The returned value must depend on the measured work—no return is needed when the work already has observable side effects.

Tags are @-prefixed tokens in the group or bench name string. They are stripped from the display name and used for filtering.

group('relation-queries @relation', () => {
  bench('ChildOf(parent)', function* () { ... });
  bench('wildcard @slow', function* () { ... }).gc(false);
});

Tags inherit: ChildOf(parent) has effective tags [@relation], wildcard has [@relation, @slow].

Filter by tag (quote the @ so pnpm passes it through):

pnpm bench "@relation"    # runs both benches
pnpm bench "@slow"        # runs only wildcard

Tags match as whole tokens at both the file and bench level, so @rel selects nothing. A run that ends up with no benches warns and is not saved.

Every run measures each benchmark in fresh-process blocks, eight by default. The first block of each benchmark runs until both the block time budget and the sample floor are met, then records its batching and sample-count decisions; later blocks replay that plan exactly. Blocks are interleaved across benchmarks so every benchmark spans the run. Inner timing samples remain useful for distributions and p99, but each block's median is treated as one independent experimental unit for comparison verdicts.

A change is flagged only when both conditions are met:

1. p ≤ alpha (two-sided Mann-Whitney U, default 0.05) — statistical significance across block medians. Comparisons with at most 50 blocks combined use the exact conditional permutation distribution of the observed ranks, including tied medians. Larger samples use a continuity- and tie-corrected normal approximation.
2. |Hodges-Lehmann Δ| ≥ minDelta (default 0.05) — practical magnitude. The relative estimator is the median of all pairwise candidate / baseline block-median ratios, minus one. Positive values mean slower and negative values mean faster.

Both gates must pass to report faster or slower; otherwise the result is neutral. The displayed Δp50 is the percent change between the two median block medians, so it can differ slightly from the Hodges-Lehmann effect used by the verdict. The Δ CI column is a rank-based interval around the Hodges-Lehmann relative effect. Its endpoints use exact tie-free Mann-Whitney critical ranks at the nominal 1 − alpha level; ties make the interval slightly conservative. The interval is uncertainty context, not a third verdict gate and not a confidence interval for Δp50.

Effect-size gating (Cliff's d) was removed: on block medians it is a monotone transform of the same U statistic behind the p-value, so a separate threshold added confusion without adding information.

A verdict must also survive the clock cross-check: when the two runs' median block clocks differ by more than 2%, the same block medians are re-judged in estimated CPU cycles (median × its block's clock probe), and a disagreement between the time and cycles verdicts skips the bench as clock-confounded rather than reporting a shift that may just be a frequency difference. When clocks are effectively equal the cross-check is inert — cycles would only re-scale time by probe jitter.

The pooled inner-sample sparklines and p99 ratio are descriptive. They help expose distribution and tail changes but are not independently significance-tested.

Labs summarizes each run's block medians with a robust relative spread: 1.4826 × MAD / median. It turns that spread into an approximate minimum detectable effect using 2.8 × spread × √(2 / blocks). The estimate is a normal-theory planning heuristic for 5% significance and 80% power with equal-sized groups, while actual verdicts use the rank test above. Treat it as an order-of-magnitude resolution diagnostic, not a guaranteed or hard detection limit.

During comparison, Labs calculates the resolution separately for baseline and candidate and displays the worse of the two. When it exceeds minDelta, the row is annotated as limited-resolution, but the verdict is still evaluated. Large, well-separated effects can therefore receive a verdict despite inconsistent fresh runs; a neutral result at limited resolution means that the data could not establish a change at that scale.

Each p-value applies to one benchmark. Labs does not currently adjust alpha across a suite, and separately saved baseline and candidate sessions can still differ in unmeasured machine state. Treat suite-wide or causal conclusions accordingly.

Place labs.config.ts alongside your bench files:

import { defineConfig } from 'labs'

export default defineConfig({
  benchDir: '.',
  benchMatch: '**/*.bench.ts',
  nodeFlags: ['--allow-natives-syntax', '--expose-gc'],
})

Sampling behavior:

• The first block of each benchmark samples until both blockTime and minSamples are met. Only measured work counts toward blockTime. Fast work is batched so each sample lasts about a millisecond, and slow work runs until the sample floor is reached even if that exceeds blockTime.
• Later blocks replay the first block's batching and sample count exactly, so every block does identical work.
• Two spreads are reported. Between-block spread of medians drives the comparison resolution and the limited-resolution annotation, evaluated against the current minDelta so changing it re-evaluates existing results. Within-block spread of samples flags noisy benchmarks whose timed work does not settle inside one process.