GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat(oracles): survivor classes with directive messages; short, streamed gap output by KyleAMathews · Pull Request #12 · TanStack/oracles

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

The gap report now tells an agent what to do next for each gap, and it never offers a way out. Its output also fits in an agent's context.

Why it matters

An agent stopped at four survivors in TanStack DB issue 2254 and recorded them as limits. The report's message gave it the exit. The message said that the mutant was either equivalent, with a maintainer's approval, or that no explored history told it apart. The maintainer decided that a message must give one clear directive and never an exit. The report also printed everything to stdout at the end: about 40,000 tokens for 155 gaps, most of it repeated text.

What changes

Survivor classes. Each survivor gets one class from what the run measured, and one directive built from the run's data:

Class What the run saw Directive
branch-never-taken An owner reached an if, but only one outcome happened Add a history that takes the other branch
not-infected The mutated comparison had the original's value in every history Add a history where the two operators disagree. The report prints the operand values seen.
not-observed The mutation changed a value, and no recorded observation changed Observe the value that this code changes, and assert on it
not-judged An observation changed, and no law rejected it Write or tighten the law that forbids the new value
indeterminate Only nondeterministic tests reached the site Make the test deterministic
dead-code No input can detect the mutant (the tag from #8) Remove the redundant code
not-distinguished The operator changes a value only on some inputs, and the capture does not show which. These operators are a loop condition, an undefined call value, a dropped field, a skipped call, and a logical operator. Add a history with the input that tells the mutant apart. The directive names that input.

A mutant with the boundary: -0 tag from #8 also gets "Add a boundary history with -0 and 0".

Failure classes (random-only-kill, agreement-only-kill, unregistered-kill, framework-error, crashed) also get directives. A crashed gap names the error and tells the author to record it as an observation and state the law.

Short standard output. stdout has these parts:

  • the totals
  • one legend line for each class that occurs
  • the gaps, grouped by function, one line each, with the owners printed once and paths relative to the package
  • the unowned warnings as one count for each function
  • the paths of summary.md and results.json

The full report stays in summary.md.

Streaming. The CLI prints the baseline result first: each oracle file's pass or fail, and the failing test with a replay line. Then it prints one line for each gap when the CLI judges it, a count every 25 mutants, and the summary. An agent can start on a gap before a long run ends.

Evidence

  • RED: a new law in the classify oracle says that every survivor has exactly one class. It also says that every message is a directive with no exit phrase. It fails on main, 11 of 11, and the 2254-shaped survivor's message is one of the failures.
  • GREEN: the fast gap oracles pass, 31,244 tests. The e2e oracle's fixed seed passes. Its model now predicts the survivor classes and checks the streamed lines. The e2e reach counts are: not observed 4, not distinguished 2, not infected 5, branch never taken 3, and indeterminate 3.
  • Streaming RED: the e2e streaming law found that a stale authored mutant was never streamed. A later commit fixed that.
  • Mutants: the oracles reject each of these. Drop the not-infected branch. Add "or equivalently" to a directive. Drop not-observed gaps from stdout. Remove the authored streaming loop.

Review round

A high-effort code review found ten problems. This PR fixes these:

  • branch-never-taken now reads the mutant's own if site. An if-true mutant changes its condition only where the condition was false, and an if-false mutant only where it was true. RED on 809c8b9: the classify oracle's new case "an if-false that always saw true" got branch-never-taken, and not-observed is owed.
  • Code that runs when a file loads the module infects every test of that file. A nondeterministic test's infection and operands judge nothing. RED on 809c8b9: the new case "a comparison infected only at module load".
  • The new not-distinguished class covers the operators whose infection the capture does not measure. RED on 809c8b9: "mutant m0: not-observed, owed not-distinguished". Mutant: if production classes those operators as not-observed, the classify oracle fails both seeds.
  • The framework-error and crashed messages name the check and the law.
  • The e2e model follows the same rules. A new witness, a cache write that no observation reads, keeps "not observed" reached, and the e2e oracle now reaches "not distinguished". The e2e fixed seed also found a model bug: the old probe guard is true under test, so its if-false and !== sites are infected. The model now says so.

Three review findings move to a follow-up PR, because each needs an oracle law first. They are the replay -t built from the suite names with regex escapes, and the 60-character cap on a gap line. The third is typed operand values: quoted strings, -0, undefined, NaN, and bigint.

Limits

  • not-infected is measured only for comparison and if mutants. The other operators infect whenever their site is reached.
  • Authored mutants are not instrumented yet, so their survivors are not-judged or not-observed.
  • The e2e oracle checks summary.md, results.json, and the streamed lines, not the exact stdout text.
  • Not in this PR: allowed(...) and the "allowed alternative" class, a provisional report on a RED baseline, the run-to-run comparison, and the daemon's streaming reply.
  • For a site that only a background reach touched, the message names "the owner tests". The classify model follows the code here. The maintainer still has to check the wording.

The squash message of #9 still describes its removed spin rule. That rule is not on main.

This pull request and its description were written by Isaac.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
KyleAMathews and others added 8 commits November 10, 2026 15:22
@KyleAMathews
test(oracles): the classify oracle law for survivor classes and direc…
454a546
…tive messages (RED)

Law: every survivor gets exactly one class from what the run measured
(not-judged, indeterminate, dead-code, branch-never-taken, not-infected,
not-observed); every gap's message is one directive from its kind's fixed
set, built from run data, with no exit; the boundary hint adds its sentence;
the stdout summary lists each gap once under its function, groups unowned
warnings by function with counts, and the streamed gap lines equal the
stdout's gap lines, each once; the kills-by-law section directs one next
step. Authority: v20 design grammar decisions 10 and 11, the v20.1
amendments (C1, C12, indeterminate), the 2026-11-10 "no outs" decision, and
the coordinator's unowned-grouping and kills-law items.

Reach: constructed histories for each new class (a comparison whose operands
never tie; the same with the boundary hint; a tie that shows nothing; a dead
logical swap; an infected survivor no observation shows; a branch with one
outcome) and the random grammar, which generates binary-swap mutants with
operand pairs, original operators, and evidence tags. The fixed-seed run
reaches the new states through constructed histories.

RED (parent, before production): `pnpm --filter ... vitest --run --config
oracle-gaps/vitest.config.mjs gap-report-classify` on origin/main: 11 of 11
fail. The parent report prints the 2254-shaped survivor with "Either the
mutant is equivalent (say why; a maintainer approves it), or ..." (scratch
check: exit phrase present: true). The calibration and the fixed-seed run
fail on the missing compressed stdout (no renderStdout), the parent's
classes, and the parent's messages.

GREEN: pending (next commit).
Mutant: pending (next commit).
Limits: the e2e oracle (gap-report-oracle) still predicts the old class
names until its model is updated; it runs only at the end, under the
fixed-seed filter.
@KyleAMathews Isaac
feat(oracles): survivor classes with directive messages, compressed s…
bc37fd4
…tdout, streamed gaps (GREEN)

Survivors get one class from what the run measured: not-judged (a changed
value no law rejected), indeterminate, dead-code (the evidence tag that no
input distinguishes the mutant), branch-never-taken (an `if` whose other
outcome no owner history took; names the outcome to add), not-infected (a
comparison whose operators agreed on every operand the owners reached;
prints those operands; the boundary tag adds "Add a boundary history with
-0 and 0."), and not-observed (infected, no observation changed; names the
infecting test). Every gap's message is one directive built from run data,
with no equivalence, approval, limit, or question. Failure classes keep
their outcomes and get directive messages.

The capture records the first operand pair of each comparison in a test
(capture-setup.mjs `cmp`), so not-infected can print the operands.

report.mjs: the full report (summary.md) keeps its shape; its unowned
warnings are grouped by function with counts; the kills-by-law section
gives one next step. renderStdout prints the totals, one legend line per
class (its directive with placeholders), every gap once under its function
(owners once per function, package-relative), unowned counts per function,
and the paths of the full report and results.json.

stream.mjs: gapLine (one form for the stream and the summary), progressLine
(a count every 25 mutants), baselineLines (each oracle file's result and a
failing law's replay). check.mjs prints the baseline, each gap as it is
judged, and returns the compressed summary.

Law: decision 11 of the v20 design grammar (survivor classes by what the CLI
measured), decision 10 (messages leave no outs), v20.1 C1 and C12, the
indeterminate class, the boundary and dead-code tags from #8, and the
coordinator's unowned-grouping and kills-law items.

Reach: gap-report-classify-oracle constructed histories for each new class,
the random grammar's binary-swap mutants with operand pairs and evidence,
and the 2254-shaped infected survivor. Reach states: not judged, branch
never taken, not infected, not infected with a boundary hint, not observed,
dead code, indeterminate, unreached.

RED (parent, 454a546's parent 1f5ac22 plus the RED oracle): 11 of 11 fail
(gap-report-classify, `pnpm vitest --run --config oracle-gaps/vitest.config.mjs
gap-report-classify`). A scratch render of the 2254-shaped survivor on the
parent prints "Either the mutant is equivalent (say why; a maintainer
approves it), or ..." and the exit check reports it.

GREEN: gap-report-classify 11/11 pass. gap-report-static, probe-form, and
mutant-validity pass with it (31,244 tests in all four fast oracles).

Mutant: (1) drop the not-infected branch in classify.mjs: the oracle fails.
(2) add "or equivalently" to the not-observed directive: fails the message
law. (3) drop not-observed gaps from renderStdout: fails the stdout law.
Each was restored byte-identical after the run.

Limits: the classify oracle emulates the capture from records, so it does
not check that capture-setup.mjs records operands from a real run; the e2e
oracle does (fixed-seed run, next commit). The model's fallback for a
background-only reach names "the owner tests" (no test infected it) and is
a choice this commit makes to match the message: flagged for the maintainer.
Authored survivors get not-observed or not-judged until authored mutants are
instrumented (C6, a later PR). The streamed-line law is checked here on the
printer and in the e2e on real runs (next commit). The allowed(...)
alternative and run-to-run change are out of scope.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(oracles): a crash's message names the error and the law it needs
7ee76ee
A crashed mutant's message said "write a law that rejects this crash", which
does not say that production threw without a law judging it. The message now
quotes the error and directs the driver: record the error as an observation
and state the law that decides if this input may throw. Failure classes keep
their outcomes; only the message changes (decision 10: no outs).

Law: decision 10 (messages leave no outs), and the coordinator's field item
from a mitt run: two crashed mutants came from a TypeError that escaped the
driver, and the agent had no directive for it.

Reach: the classify oracle's constructed crash history and the random
grammar's crash effect.

RED: the previous report.mjs (the commit before this one) with the new
directive regex in the oracle: 2 of 11 fail (`crashed` message mismatch).
GREEN: 11 of 11 pass.

Mutant: the old wording ("write a law ... rejects this crash") fails the new
directive law, as the RED shows; a message with an exit word is rejected by
the exit law.

Limits: the message quotes the first line of the error, at most 120
characters; a longer error is truncated in the message and whole in the
report's Detail line.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(oracles): stream every authored gap, and name an unapplied mutant…
44b7f04
…'s line

The fixed-seed e2e streaming law (added in the next commit) found two gaps in
the streamed lines of the first version of this change:

1. An authored mutant whose `find` no longer matches is `does-not-apply`, and
   the authored loop skipped it before its streamed line. It is now streamed
   once, after the authored phase, with every other authored gap.
2. Such a mutant has no line, so its gap line read `Lundefined`. It now reads
   `L?`, which says the line is unknown.

The legend's crash directive now shows the error as a placeholder instead of
"(no message)".

Law: the streaming law of the gap-report e2e oracle (law 11): the streamed gap
lines are the report's gaps, each printed once, none missing. Authority: the
2026-11-10 maintainer decision that results stream as they are judged, and
decision 10 (one clear line per gap).

RED: the e2e fixed-seed run on the previous commit (bc37fd4) with the
streaming law: "the streamed gap lines are not the report's gaps: 0 streamed,
1 reported", on a history with a stale authored mutant (the first run, and
a second run with the `Lundefined` line still in place).
GREEN: the e2e fixed-seed run after this change: 1 passed, 12 skipped (13).
Mutant: removing the authored streaming loop fails the same law (the RED
above).

Limits: the streaming law is checked on the cold CLI's stderr; the daemon
streams its reply's stderr to the same lines, which the e2e's law 9 compares
only through the report, not the stream. The streamed-line law is checked
only in the fixed-seed subset.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(oracles): the gap-report e2e model predicts the survivor classes…
0fa3b76
… and checks streaming

The model: survivors are not-judged (a changed observation), indeterminate,
branch-never-taken (an `if` whose line one outcome never took), not-infected
(a comparison no owner run infected; the interpreter records the infected
comparison sites, including the loop and until comparisons), and not-observed
(infected, no observation changed). Authored survivors are not-judged or
not-observed, since authored mutants are not instrumented.

The e2e check adds law 11, streaming: the gap lines the CLI streams to stderr
as each mutant is judged are the report's gaps, each once. The class names in
the reach states and the law's prose follow the same vocabulary.

Law: decision 11 of the v20 design grammar (survivor classes by what the CLI
measured), and the streaming decision of 2026-11-10 (results stream as they
are judged).

Reach (fixed-seed run): branch never taken 3, not infected 5, not judged 4,
not observed 4 histories, and the streamed-line law checked on every history
that the e2e runs.

RED: the e2e model before this commit predicted generator-gap and
comparison-gap, and the CLI reports the new classes, so the fixed-seed run
could not pass (the first run in the previous commit's RED).
GREEN: fixed-seed run: 1 passed, 12 skipped (13), 186.9 s, under the shared
e2e lock.
Mutant: the streaming law rejects a CLI that omits an authored gap (the
previous commit's RED).

Limits: the infection model covers the comparison sites the interpreter runs
(conditions, loops, until). The e2e fixtures generate no dead-code or
boundary-hint evidence, so those classes are covered by the classify oracle
only. Only the fixed-seed subset ran; the full e2e runs in CI.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews
Merge remote-tracking branch 'origin/main' into feat/survivor-classes…
fd10aa8
…-report

Co-authored-by: Isaac <[email protected]>

# Conflicts:
#	packages/oracles/oracle-gaps/check.mjs
#	packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs
#	packages/oracles/oracle-gaps/report.mjs
@KyleAMathews Isaac
chore: changeset for the gap report's directives and output
526d919
Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
chore: shorten a changeset sentence
809c8b9
Co-authored-by: Isaac <[email protected]>
@changeset-bot

changeset-bot Bot commented Nov 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b467f32

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@tanstack/oracles Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

  • X
  • Mastodon
  • Reddit
  • LinkedIn
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pkg-pr-new

pkg-pr-new Bot commented Nov 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@12
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@12

commit: b467f32

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

KyleAMathews and others added 3 commits November 10, 2026 18:49
@KyleAMathews Isaac
fix(oracles): classify survivors from the mutant's own site; add not-…
7e55966
…distinguished

Review round on the survivor classes:

- branch-never-taken reads the mutant's own `if` site: an if-true changes
  its condition only where it was false, an if-false only where it was
  true, and the class applies only when that infection is absent (the
  sibling lookup is gone). RED on 809c8b9: the classify oracle's
  constructed case "an if-false that always saw true" got
  branch-never-taken where not-observed is owed.
- Code that runs when a file loads the module infects every test of that
  file, and a nondeterministic test's infection and operands judge
  nothing. RED on 809c8b9: the constructed case "a comparison infected
  only at module load".
- New class `not-distinguished` for the operators whose infection the
  capture does not measure (a loop's condition, an undefined call value, a
  dropped field, a skipped call, a logical operator). Its directive names
  the input that distinguishes the mutant. RED on 809c8b9: "mutant m0:
  not-observed, owed not-distinguished". Mutant: classing those operators
  as not-observed fails the classify oracle on both seeds.
- framework-error and crashed messages name the check and the law; the
  dead-code header is exempt from the directive rule.
- The e2e model follows the same rule: its unmeasured set matches
  classify.mjs, and the old probe guard's if-false and `!==` sites are
  infected (the guard is true under test). A new witness (a cache write no
  observation reads) keeps "not observed" reached, and the e2e oracle now
  reaches "not distinguished".

GREEN: fast gap oracles 31,244 of 31,244; e2e fixed seed passes (reach:
not observed 4, not distinguished 2, not infected 5, branch never taken
3, indeterminate 3); tsc and eslint clean.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews
Merge remote-tracking branch 'origin/main' into feat/survivor-classes…
126fd98
…-report
@KyleAMathews Isaac
chore(oracles): drop the last law.owner lines in untyped oracle files
b467f32
#11 removed law.owner; two .js/.mjs oracle files that the typecheck does
not read kept the field.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews
KyleAMathews merged commit 5331fbd into main Nov 11, 2026
8 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@KyleAMathews
KyleAMathews deleted the feat/survivor-classes-report branch November 11, 2026 02:39
KyleAMathews added a commit that referenced this pull request Nov 11, 2026
@KyleAMathews Isaac
docs(oracles): write the general law down, and trigger the design gra…
573c671
…mmar

Agents go from examples to a general law only when made to write it
down, and no agent in three validation rounds opened the design-grammar
card, because it triggered on judgment ("a hard law").

- Design step 1: before the law sheet, each law gets a written block:
  the examples with sources, what varies and what stays, the law as one
  "For every ..." sentence with its edges, a new case it predicts, and
  every example rebuilt from the sentence alone.
- Five concrete triggers send the agent to the design-grammar card; the
  evidence says which fired.
- Review: "Law as examples" and "Skipped design grammar" findings.
- Evaluate-review: the causal condition is written as a sentence over
  histories, with a second history it predicts.
- Setup: TypeScript 7 is an environment limit (typescript-eslint), and
  the plugin's `unparsedModule` report for a `.js` oracle file that
  imports TypeScript (#13).
- Design: the interim exit-wording bullet is gone with #12; the rule not
  to write those words in a report stays.

Co-authored-by: Isaac <[email protected]>
This was referenced Nov 11, 2026
feat(oracles): every gap CLI run ends in a report: refused (exit 2) or internal error (exit 3), never a stack trace #14
Merged
fix(oracles): working replay -t, one-line gap lines, typed operands, and a two-step not-observed directive #20
Merged
docs(oracles): ship setup, design, and review skills, with a prompt to try them #7
Open
test(oracles): fix the e2e gap oracle's determinism and summary-calibration tests #28
Merged
KyleAMathews added a commit that referenced this pull request Nov 11, 2026
@KyleAMathews Isaac
test(oracles): fix the e2e gap oracle's determinism and summary-calib…
336c811
…ration tests (#28)

test(oracle-gaps): make the e2e determinism test and the summary calibration pass

namedScratch now names the longest path first, because the scratch package
lies inside the CLI's directory, and names the runner's process id in its
wrapper file names. Both random ids reached the recorded trace of a refused
run, so determinism failed.

The "summary omits a gap" calibration now drops the gap from the summary and
from the stream. The streaming check (#12) caught the old corruption first,
so it never reached report completeness, the law it names.

Co-authored-by: Isaac <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@KyleAMathews
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.