GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||
GeezerVison is showing github.com — feat(oracles): survivor classes with directive messages; short, streamed gap output by KyleAMathews · Pull Request #12 · TanStack/oracles Conversation
|
| Class | What the run saw | Directive |
|---|---|---|
branch-never-taken |
An owner reached an if, but only one outcome happened |
Add a history that takes the other branch |
not-infected |
The mutated comparison had the original's value in every history | Add a history where the two operators disagree. The report prints the operand values seen. |
not-observed |
The mutation changed a value, and no recorded observation changed | Observe the value that this code changes, and assert on it |
not-judged |
An observation changed, and no law rejected it | Write or tighten the law that forbids the new value |
indeterminate |
Only nondeterministic tests reached the site | Make the test deterministic |
dead-code |
No input can detect the mutant (the tag from #8) | Remove the redundant code |
not-distinguished |
The operator changes a value only on some inputs, and the capture does not show which. These operators are a loop condition, an undefined call value, a dropped field, a skipped call, and a logical operator. |
Add a history with the input that tells the mutant apart. The directive names that input. |
A mutant with the boundary: -0 tag from #8 also gets "Add a boundary history with -0 and 0".
Failure classes (random-only-kill, agreement-only-kill, unregistered-kill, framework-error, crashed) also get directives. A crashed gap names the error and tells the author to record it as an observation and state the law.
Short standard output. stdout has these parts:
summary.md and results.jsonThe full report stays in summary.md.
Streaming. The CLI prints the baseline result first: each oracle file's pass or fail, and the failing test with a replay line. Then it prints one line for each gap when the CLI judges it, a count every 25 mutants, and the summary. An agent can start on a gap before a long run ends.
main, 11 of 11, and the 2254-shaped survivor's message is one of the failures.not-infected branch. Add "or equivalently" to a directive. Drop not-observed gaps from stdout. Remove the authored streaming loop.A high-effort code review found ten problems. This PR fixes these:
branch-never-taken now reads the mutant's own if site. An if-true mutant changes its condition only where the condition was false, and an if-false mutant only where it was true. RED on 809c8b9: the classify oracle's new case "an if-false that always saw true" got branch-never-taken, and not-observed is owed.not-distinguished class covers the operators whose infection the capture does not measure. RED on 809c8b9: "mutant m0: not-observed, owed not-distinguished". Mutant: if production classes those operators as not-observed, the classify oracle fails both seeds.framework-error and crashed messages name the check and the law.if-false and !== sites are infected. The model now says so.Three review findings move to a follow-up PR, because each needs an oracle law first. They are the replay -t built from the suite names with regex escapes, and the 60-character cap on a gap line. The third is typed operand values: quoted strings, -0, undefined, NaN, and bigint.
not-infected is measured only for comparison and if mutants. The other operators infect whenever their site is reached.not-judged or not-observed.summary.md, results.json, and the streamed lines, not the exact stdout text.allowed(...) and the "allowed alternative" class, a provisional report on a RED baseline, the run-to-run comparison, and the daemon's streaming reply.The squash message of #9 still describes its removed spin rule. That rule is not on main.
This pull request and its description were written by Isaac.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…tive messages (RED) Law: every survivor gets exactly one class from what the run measured (not-judged, indeterminate, dead-code, branch-never-taken, not-infected, not-observed); every gap's message is one directive from its kind's fixed set, built from run data, with no exit; the boundary hint adds its sentence; the stdout summary lists each gap once under its function, groups unowned warnings by function with counts, and the streamed gap lines equal the stdout's gap lines, each once; the kills-by-law section directs one next step. Authority: v20 design grammar decisions 10 and 11, the v20.1 amendments (C1, C12, indeterminate), the 2026-11-10 "no outs" decision, and the coordinator's unowned-grouping and kills-law items. Reach: constructed histories for each new class (a comparison whose operands never tie; the same with the boundary hint; a tie that shows nothing; a dead logical swap; an infected survivor no observation shows; a branch with one outcome) and the random grammar, which generates binary-swap mutants with operand pairs, original operators, and evidence tags. The fixed-seed run reaches the new states through constructed histories. RED (parent, before production): `pnpm --filter ... vitest --run --config oracle-gaps/vitest.config.mjs gap-report-classify` on origin/main: 11 of 11 fail. The parent report prints the 2254-shaped survivor with "Either the mutant is equivalent (say why; a maintainer approves it), or ..." (scratch check: exit phrase present: true). The calibration and the fixed-seed run fail on the missing compressed stdout (no renderStdout), the parent's classes, and the parent's messages. GREEN: pending (next commit). Mutant: pending (next commit). Limits: the e2e oracle (gap-report-oracle) still predicts the old class names until its model is updated; it runs only at the end, under the fixed-seed filter.
…tdout, streamed gaps (GREEN) Survivors get one class from what the run measured: not-judged (a changed value no law rejected), indeterminate, dead-code (the evidence tag that no input distinguishes the mutant), branch-never-taken (an `if` whose other outcome no owner history took; names the outcome to add), not-infected (a comparison whose operators agreed on every operand the owners reached; prints those operands; the boundary tag adds "Add a boundary history with -0 and 0."), and not-observed (infected, no observation changed; names the infecting test). Every gap's message is one directive built from run data, with no equivalence, approval, limit, or question. Failure classes keep their outcomes and get directive messages. The capture records the first operand pair of each comparison in a test (capture-setup.mjs `cmp`), so not-infected can print the operands. report.mjs: the full report (summary.md) keeps its shape; its unowned warnings are grouped by function with counts; the kills-by-law section gives one next step. renderStdout prints the totals, one legend line per class (its directive with placeholders), every gap once under its function (owners once per function, package-relative), unowned counts per function, and the paths of the full report and results.json. stream.mjs: gapLine (one form for the stream and the summary), progressLine (a count every 25 mutants), baselineLines (each oracle file's result and a failing law's replay). check.mjs prints the baseline, each gap as it is judged, and returns the compressed summary. Law: decision 11 of the v20 design grammar (survivor classes by what the CLI measured), decision 10 (messages leave no outs), v20.1 C1 and C12, the indeterminate class, the boundary and dead-code tags from #8, and the coordinator's unowned-grouping and kills-law items. Reach: gap-report-classify-oracle constructed histories for each new class, the random grammar's binary-swap mutants with operand pairs and evidence, and the 2254-shaped infected survivor. Reach states: not judged, branch never taken, not infected, not infected with a boundary hint, not observed, dead code, indeterminate, unreached. RED (parent, 454a546's parent 1f5ac22 plus the RED oracle): 11 of 11 fail (gap-report-classify, `pnpm vitest --run --config oracle-gaps/vitest.config.mjs gap-report-classify`). A scratch render of the 2254-shaped survivor on the parent prints "Either the mutant is equivalent (say why; a maintainer approves it), or ..." and the exit check reports it. GREEN: gap-report-classify 11/11 pass. gap-report-static, probe-form, and mutant-validity pass with it (31,244 tests in all four fast oracles). Mutant: (1) drop the not-infected branch in classify.mjs: the oracle fails. (2) add "or equivalently" to the not-observed directive: fails the message law. (3) drop not-observed gaps from renderStdout: fails the stdout law. Each was restored byte-identical after the run. Limits: the classify oracle emulates the capture from records, so it does not check that capture-setup.mjs records operands from a real run; the e2e oracle does (fixed-seed run, next commit). The model's fallback for a background-only reach names "the owner tests" (no test infected it) and is a choice this commit makes to match the message: flagged for the maintainer. Authored survivors get not-observed or not-judged until authored mutants are instrumented (C6, a later PR). The streamed-line law is checked here on the printer and in the e2e on real runs (next commit). The allowed(...) alternative and run-to-run change are out of scope. Co-authored-by: Isaac <[email protected]>
A crashed mutant's message said "write a law that rejects this crash", which
does not say that production threw without a law judging it. The message now
quotes the error and directs the driver: record the error as an observation
and state the law that decides if this input may throw. Failure classes keep
their outcomes; only the message changes (decision 10: no outs).
Law: decision 10 (messages leave no outs), and the coordinator's field item
from a mitt run: two crashed mutants came from a TypeError that escaped the
driver, and the agent had no directive for it.
Reach: the classify oracle's constructed crash history and the random
grammar's crash effect.
RED: the previous report.mjs (the commit before this one) with the new
directive regex in the oracle: 2 of 11 fail (`crashed` message mismatch).
GREEN: 11 of 11 pass.
Mutant: the old wording ("write a law ... rejects this crash") fails the new
directive law, as the RED shows; a message with an exit word is rejected by
the exit law.
Limits: the message quotes the first line of the error, at most 120
characters; a longer error is truncated in the message and whole in the
report's Detail line.
Co-authored-by: Isaac <[email protected]>
…'s line The fixed-seed e2e streaming law (added in the next commit) found two gaps in the streamed lines of the first version of this change: 1. An authored mutant whose `find` no longer matches is `does-not-apply`, and the authored loop skipped it before its streamed line. It is now streamed once, after the authored phase, with every other authored gap. 2. Such a mutant has no line, so its gap line read `Lundefined`. It now reads `L?`, which says the line is unknown. The legend's crash directive now shows the error as a placeholder instead of "(no message)". Law: the streaming law of the gap-report e2e oracle (law 11): the streamed gap lines are the report's gaps, each printed once, none missing. Authority: the 2026-11-10 maintainer decision that results stream as they are judged, and decision 10 (one clear line per gap). RED: the e2e fixed-seed run on the previous commit (bc37fd4) with the streaming law: "the streamed gap lines are not the report's gaps: 0 streamed, 1 reported", on a history with a stale authored mutant (the first run, and a second run with the `Lundefined` line still in place). GREEN: the e2e fixed-seed run after this change: 1 passed, 12 skipped (13). Mutant: removing the authored streaming loop fails the same law (the RED above). Limits: the streaming law is checked on the cold CLI's stderr; the daemon streams its reply's stderr to the same lines, which the e2e's law 9 compares only through the report, not the stream. The streamed-line law is checked only in the fixed-seed subset. Co-authored-by: Isaac <[email protected]>
… and checks streaming The model: survivors are not-judged (a changed observation), indeterminate, branch-never-taken (an `if` whose line one outcome never took), not-infected (a comparison no owner run infected; the interpreter records the infected comparison sites, including the loop and until comparisons), and not-observed (infected, no observation changed). Authored survivors are not-judged or not-observed, since authored mutants are not instrumented. The e2e check adds law 11, streaming: the gap lines the CLI streams to stderr as each mutant is judged are the report's gaps, each once. The class names in the reach states and the law's prose follow the same vocabulary. Law: decision 11 of the v20 design grammar (survivor classes by what the CLI measured), and the streaming decision of 2026-11-10 (results stream as they are judged). Reach (fixed-seed run): branch never taken 3, not infected 5, not judged 4, not observed 4 histories, and the streamed-line law checked on every history that the e2e runs. RED: the e2e model before this commit predicted generator-gap and comparison-gap, and the CLI reports the new classes, so the fixed-seed run could not pass (the first run in the previous commit's RED). GREEN: fixed-seed run: 1 passed, 12 skipped (13), 186.9 s, under the shared e2e lock. Mutant: the streaming law rejects a CLI that omits an authored gap (the previous commit's RED). Limits: the infection model covers the comparison sites the interpreter runs (conditions, loops, until). The e2e fixtures generate no dead-code or boundary-hint evidence, so those classes are covered by the classify oracle only. Only the fixed-seed subset ran; the full e2e runs in CI. Co-authored-by: Isaac <[email protected]>
…-report Co-authored-by: Isaac <[email protected]> # Conflicts: # packages/oracles/oracle-gaps/check.mjs # packages/oracles/oracle-gaps/gap-report-oracle.property.test.mjs # packages/oracles/oracle-gaps/report.mjs
Co-authored-by: Isaac <[email protected]>
Co-authored-by: Isaac <[email protected]>
There was an error while loading. Please reload this page.
🦋 Changeset detectedLatest commit: b467f32 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Pre-merge checks:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
commit: |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…distinguished Review round on the survivor classes: - branch-never-taken reads the mutant's own `if` site: an if-true changes its condition only where it was false, an if-false only where it was true, and the class applies only when that infection is absent (the sibling lookup is gone). RED on 809c8b9: the classify oracle's constructed case "an if-false that always saw true" got branch-never-taken where not-observed is owed. - Code that runs when a file loads the module infects every test of that file, and a nondeterministic test's infection and operands judge nothing. RED on 809c8b9: the constructed case "a comparison infected only at module load". - New class `not-distinguished` for the operators whose infection the capture does not measure (a loop's condition, an undefined call value, a dropped field, a skipped call, a logical operator). Its directive names the input that distinguishes the mutant. RED on 809c8b9: "mutant m0: not-observed, owed not-distinguished". Mutant: classing those operators as not-observed fails the classify oracle on both seeds. - framework-error and crashed messages name the check and the law; the dead-code header is exempt from the directive rule. - The e2e model follows the same rule: its unmeasured set matches classify.mjs, and the old probe guard's if-false and `!==` sites are infected (the guard is true under test). A new witness (a cache write no observation reads) keeps "not observed" reached, and the e2e oracle now reaches "not distinguished". GREEN: fast gap oracles 31,244 of 31,244; e2e fixed seed passes (reach: not observed 4, not distinguished 2, not infected 5, branch never taken 3, indeterminate 3); tsc and eslint clean. Co-authored-by: Isaac <[email protected]>
#11 removed law.owner; two .js/.mjs oracle files that the typecheck does not read kept the field. Co-authored-by: Isaac <[email protected]>
There was an error while loading. Please reload this page.
…mmar
Agents go from examples to a general law only when made to write it
down, and no agent in three validation rounds opened the design-grammar
card, because it triggered on judgment ("a hard law").
- Design step 1: before the law sheet, each law gets a written block:
the examples with sources, what varies and what stays, the law as one
"For every ..." sentence with its edges, a new case it predicts, and
every example rebuilt from the sentence alone.
- Five concrete triggers send the agent to the design-grammar card; the
evidence says which fired.
- Review: "Law as examples" and "Skipped design grammar" findings.
- Evaluate-review: the causal condition is written as a sentence over
histories, with a second history it predicts.
- Setup: TypeScript 7 is an environment limit (typescript-eslint), and
the plugin's `unparsedModule` report for a `.js` oracle file that
imports TypeScript (#13).
- Design: the interim exit-wording bullet is gone with #12; the rule not
to write those words in a report stays.
Co-authored-by: Isaac <[email protected]>
…ration tests (#28) test(oracle-gaps): make the e2e determinism test and the summary calibration pass namedScratch now names the longest path first, because the scratch package lies inside the CLI's directory, and names the runner's process id in its wrapper file names. Both random ids reached the recorded trace of a refused run, so determinism failed. The "summary omits a gap" calibration now drops the gap from the summary and from the stream. The streaming check (#12) caught the old corruption first, so it never reached report completeness, the law it names. Co-authored-by: Isaac <[email protected]>
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.