GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(eslint-plugin-oracles): rules never throw, and oracle rules cannot be disabled inline by KyleAMathews · Pull Request #6 · TanStack/oracles

Conversation

@KyleAMathews

@KyleAMathews KyleAMathews commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

The oracle lint rules now hold their two promises. They never throw, and a comment in the file cannot silence them. Before this change, one bad setting made oracle/oracle-property-only stop the ESLint run. Also, an eslint-disable comment could turn off any oracle rule, and nothing reported it.

Why it matters

An agent reads lint output after each edit. A rule that throws stops the run, so the agent gets no report for any rule in that file. A rule that a comment can switch off gives the agent an easy exit: it disables the rule and the report goes away. Both behaviors break the decision that oracle lint is mechanical and cannot be bypassed.

What changes

The rules do not throw. oracle/oracle-property-only reports a malformed legacyCampaigns entry as a lint message. It also reports an implementationModules or replayWitnessModules setting that is not an array. Before, it threw "Error while loading rule" (the behavior came with #4). The message sits at line 1, column 0 under every parser.

Oracle files ignore inline config. The recommended config sets linterOptions.noInlineConfig for files whose name contains oracle. ESLint has no setting for each rule, so this applies to every rule in those files. A directive in an oracle file has no effect, and ESLint reports that it ignored it.

A new rule reports directives that would silence an oracle rule. oracle/no-oracle-disable is on in recommended and production. It reads the directives that ESLint parsed (sourceCode.getDisableDirectives() and getInlineConfigNodes()), so it does not depend on how the comment is spelled. It reports these:

  • a disable directive that names an oracle/* rule, quoted or not, in any of the line, next-line, or block forms.
  • a disable directive with no rule list, because it silences every rule.
  • an inline config comment such as /* eslint oracle/probe-form: "warn" */ that changes an oracle rule to any severity.

A directive can hide a report on the lines that it covers. This includes a directive that names this rule or has no rule list. The rule reports such a directive at the first token of the file. The message gives the line of the directive.

// eslint-disable-next-line "oracle/probe-form"
globalThis.__oracleProbe?.('x', 1)
// now reported: the quoted rule name no longer hides the directive

Probe-form also checks single-package repositories. The production config used the glob packages/*/src/**, so a repository with src/ at the root got no probe-form messages. The glob now includes src/**.

Limits

These cases stay open. The rule header, the README, and the changeset document them:

  • A directive on the first line of a file that names this rule or has no rule list covers the start of the file. ESLint hides its report, and no earlier position exists for it.
  • In production source, an inline config comment that turns off oracle/no-oracle-disable stops the rule for that file. In oracle files, noInlineConfig stops that comment.
  • oracle/no-disabled and oracle/replayable-seed accept a setting that is not an array without a throw, but they do not report it yet. A follow-up will report malformed settings in every rule.

How the oracles check it

Each fix came after its oracle extension. The test commit is RED at its tip, and the fix commit makes it GREEN. Each fix commit has a hostile mutant that its oracle rejects. The commit messages contain the evidence blocks.

  • No throws. Five oracle drivers check the law "a rule never throws on a program that parses". They use one shared helper. It counts a failure against the rule only when ESLint attaches a ruleId to the error. It passes all other errors through, so a bug in the test harness does not count as a rule failure.
  • Directives. The no-oracle-disable oracle generates each directive form, with and without quoted names. It generates rule lists that name oracle rules, other rules, or both. It also generates descriptions, config comments, self-naming lists, blanket directives, and comments that only mention oracle/. Each case occurs in oracle files and in production source.
  • Settings and calibrations. The oracle-property-only oracle generates malformed settings and files that start with a comment. The probe-form oracle generates the single-package layout. Its calibrations now skip a trace that made the rule throw.

Verification

  • Plugin suite: 15 files, 4,296 tests pass.
  • tsc, ESLint, and Prettier pass.
  • Review: a high-effort code review (10 findings) and a simplification pass (17 suggestions) had an evaluation with a ledger. The ledger shows 17 fixed, 1 accepted design, 11 refuted, 2 already fixed, and 7 open limits. The open limits are the settings item above and five simplifications that we did not apply.

Merge with squash: the test commits are RED at their tips on purpose.

This pull request and its description were written by Isaac.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
KyleAMathews and others added 23 commits November 10, 2026 10:32
@KyleAMathews Isaac
test(eslint-plugin-oracles): a rule that throws fails its oracles' pr…
d4df58f
…edictions

A rule that throws aborts ESLint's lint of the file, so the error is not a
judged outcome. The four oracles that lint a rule now record a throw as an
observation, and the predicted reports reject it.

Law: a rule never throws on a program that parses.
Authority: P8 of the frozen v20 design grammar
(~/.cache/oracle-grammar/v20-design-grammar/freeze.md) and the maintainer
decision of 2026-11-09, "lint rules never throw". AGENTS.md "Lint feedback
while you work" needs the hooks to report on every edit, which a throw breaks.

Oracles extended: probe-form (lintWithProductionConfig), fixed-seed
(lintProgram: no-disabled and replayable-seed), replay-observation (reports),
oracle-property-only (lintRendered).

Reach: the law's states are the programs each driver lints. The
oracle-property-only grammar reaches the throwing state through its malformed
entries (bareString, fileNotString, reasonNotString, outsideGlob,
implementationModule); the other three drivers reach it through their
generated programs.

RED (parent e1fb504 rules, this commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/oracle-property-only-oracle.property.test.js
  2 of 5 tests fail (fixed seed 20261109, random seed): "the rule threw, which
  aborts ESLint's run of the file: ... Error while loading rule
  'oracle/oracle-property-only': ... legacyCampaigns entry 0 is not an oracle file".
  This RED is intentional: the next commit changes the rule and the model.

GREEN (this commit, probe-form, fixed-seed, replay-observation unchanged on
parent rules): all four test files pass except the oracle-property-only
bad-entry tests above, which the next commit turns green.

Mutant: a hostile `throw` at the top of probe-form's and no-disabled's create
for oracle files. probe-form's four checked outcomes and no-disabled's fail
with `threw: Error while loading rule 'oracle/probe-form': hostile mutant:
rule threw`. Before this commit the same mutant ended the run with the thrown
error instead of a checked outcome. Calibration: a throw injected into a clean
history of oracle-property-only is rejected.

Limits: a rule that never returns (a hang) is not covered. The oracle-property-only
bad-entry tests stay red until the rule reports them. Part-reach oracles
(no-unit-test) do not lint their drivers through a throw-observing path yet.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): oracle-property-only reports a bad legac…
9c159fa
…yCampaigns entry

The rule throws when it reads a malformed or unjudgeable legacyCampaigns
entry, so ESLint aborts the file with "Error while loading rule". The model
owed a configuration error for those entries. Under the never-throws law the
owed outcome is a lint report, so the oracle now expects one, and it names
the rule's report for each kind of bad entry.

Law: a rule never throws on a program that parses. A malformed entry, or a
legacyCampaigns value that is not an array, is reported as malformedEntry or
malformedList at line 1; an entry that names a file outside the oracle glob
(including README.md) or an implementation module is reported as refusedEntry.
A reported entry exempts nothing; the file's other entries are judged as usual.
Authority: P8 of the frozen v20 design grammar and the maintainer decision of
2026-11-09, "lint rules never throw".

Grammar extended: entry kinds readme, missingFile, fileNotString, notAList
(bareString, reasonNotString, outsideGlob, implementationModule were already
generated). Constructed histories added for a README entry and a non-array list.
The model's expected outcome is now a report list; the configuration-error
branch is removed.

Reach: each new state is entered by a constructed history (README entry,
refused entry, malformed entry, non-array list) and by the generator's
program (60 runs, fixed seed 20261109).

RED (parent: the rule from e1fb504, this commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/oracle-property-only-oracle.property.test.js
  2 of 5 fail (fixed seed 20261109, random seed). First counterexample:
  {"entry":"outsideGlob", ...}; difference: "the rule threw, which aborts
  ESLint's run of the file: a rule never throws on a program that parses".
  The rule throws for every bad-entry kind, so each constructed history fails.

GREEN: the next commit changes the rule.

Mutant: the parent rule itself is the hostile mutant (it throws on every
malformed or refused entry). The calibration "a rule's throw in a clean
history" replaces a clean outcome with a throw and the check rejects it.

Limits: implementationModules, which the rule also reads, is not generated;
a non-array implementationModules still throws until a later change covers it.
A rule that never returns is not covered.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): oracle-property-only reports a bad legacy…
cff14de
…Campaigns entry, never throws

A malformed or unjudgeable legacyCampaigns entry made the rule throw while it
read the settings, so ESLint stopped with "Error while loading rule" and the
hooks reported nothing for the file. The rule now reports the entry instead.

Law: a rule never throws on a program that parses (P8 of the frozen v20 design
grammar; maintainer decision 2026-11-09). Reported entries exempt nothing; the
file's other entries are judged as usual.

Change:
- malformed entry -> malformedEntry at line 1
- legacyCampaigns not an array -> malformedList at line 1 (replaces the file's list)
- outside the oracle glob (including README.md) or an implementation module ->
  refusedEntry at line 1
ESLint runs a rule once per file, so each oracle file that the configuration
reaches reports the entry. README and the changeset text that said "the rule
throws" now describe the reports.

Reach: the extended oracle (previous commit) reaches each state: refused,
malformed, and non-array lists, in 60 generated runs (fixed seed 20261109) and
in constructed histories for a README entry and a non-array list.

RED (parent: rule at e1fb504 with the previous commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/oracle-property-only-oracle.property.test.js
  2 of 5 fail (fixed seed 20261109, random seed); first counterexample
  entry "outsideGlob", difference "the rule threw, which aborts ESLint's run of
  the file".

GREEN (this commit): the same command, 5 of 5 pass, every state reached (the
oracle prints the reach counts for both seeds).

Mutant: (a) drop the malformed entry's report: 2 of 5 fail, difference "the
observed value differs from the owed value". (b) refused entry still judged
(exempts): 2 of 5 fail, same difference. Both mutants were run on the fixed
rule and then restored.

Limits: implementationModules is not generated; a non-array implementationModules
still throws. A rule that never returns (a hang) is not covered. The entry
checks cannot tell a new baseline entry from an old one (unchanged).

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): a disable directive does not silence the…
7754634
… oracle rule

P8 of the frozen v20 design grammar: lint rules cannot be disabled inline. The
oracle-property-only oracle had no directive in its grammar, so a directive
that silenced the rule was never generated. The grammar now writes a block
directive on the alias import's line, which covers every site below it:
`/* eslint-disable oracle/oracle-property-only */` or a bare `/* eslint-disable */`.
The law: the reports are the same with or without the directive.

Authority: P8 (frozen v20 design grammar, lint rules cannot be disabled inline),
and the maintainer decision of 2026-11-09 that the oracle rules are mechanical.

Grammar extended: directive kinds none, oracleBlock, allBlock. The state
"a disable directive is written over a reported site, and it does not silence
it" is measured and has a constructed history (a block directive over a raw
campaign). The model is unchanged: the expected reports ignore the directive.

Reach: the constructed history plus the generator (60 runs, fixed seed
20261109). The one-line form (eslint-disable-next-line) is not generated because
it moves the site lines; the block forms are generated.

RED (parent: the rule and config from e1fb504 plus the earlier fixes):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/oracle-property-only-oracle.property.test.js
  2 of 5 fail (fixed seed 20261109, random seed). First counterexample:
  {"directive":"oracleBlock", "sites":[{"kind":"prop","root":"fcTest"}]};
  difference "the observed value differs from the owed value": the block
  directive silenced the rule.

GREEN: the next commit sets noInlineConfig on the recommended config.

Mutant: the next commit's config with `noInlineConfig` removed fails the same
two tests (difference "the observed value differs from the owed value").

Limits: a one-line directive is not generated. The production config (probe-form
on production source) is not covered by this law: it still honors directives,
pending a maintainer decision (see the next commit's message).

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): recommended config ignores inline disable…
366f42c
…s of oracle rules

P8 (frozen v20 design grammar): lint rules cannot be disabled inline. A
`// eslint-disable-next-line oracle/oracle-property-only` (or a block directive)
silenced the rule in any oracle file, so the rule was not enforced where the
directive sat. The `recommended` config now sets `linterOptions.noInlineConfig`.
ESLint then ignores the directive and warns that it has no effect.

Side effect, stated: ESLint has no per-rule switch, so in an oracle file the
setting also ignores directives for other rules. ESLint reports each ignored
directive as a warning. The repository's own directive for no-sparse-arrays in
packages/oracles/tests/oracle-property-scheduler-oracle.property.test.ts is in an
oracle file, so it was replaced with an equivalent construction (a hole at index
0 made by Object.assign on new Array(2)).

Not changed: the `production` config (probe-form on production source). Setting
noInlineConfig there would also ignore every other rule's directive in production
source, which is a maintainer decision, so probe-form can still be disabled inline
in production files. Recorded as a limit.

Law: in an oracle file a disable directive does not silence an oracle rule (the
reports are the same with or without it).
Authority: P8, frozen v20 design grammar.

Reach and RED: the previous commit's oracle (directive dimension, constructed
block-directive history, 60 generated runs at fixed seed 20261109).

RED (parent: config without noInlineConfig, this commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/oracle-property-only-oracle.property.test.js
  2 of 5 fail, difference "the observed value differs from the owed value".

GREEN (this commit): the same command, 5 of 5 pass, with every state reached.
Full plugin suite: 14 files, 4290 tests pass. The scheduler oracle in packages/oracles
passes (7 of 7) after the directive replacement.

Mutant: the config with noInlineConfig removed (the parent) fails the same two
tests: the block directive silences the rule.

Limits: production config not covered (see above); a one-line directive
(eslint-disable-next-line) is not generated by the oracle.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): probe-form covers a single-package layout
5f1e006
The production config's files glob is packages/*/src/**, so a single-package
repository with src/ at its root gets no probe-form reports. Production source
is src under a package or at the root of a single-package repository, and the
oracle must generate both layouts.

Law: a state probe in production source is reported exactly where the form is
broken, in either layout; the root config enables the rule for production
source alone. Authority: the probe-form law's existing authority (the
maintainer's decisions of 2026-11-08 on the probe form) and P8 of the frozen
v20 design grammar for the production config's coverage of the rule.

Grammar extended: programArbitrary draws a layout (monorepo or singlePackage);
the drivers lint the program under packages/oracles/src/ (monorepo) or src/ at
the root (singlePackage). The census gains a single-package history, and the
state "single-package layout" is measured, with a constructed history.

Reach: the constructed history plus the generator (fixed seed 2234, 40 runs,
and the census). The layout does not change the model's reports, only the file
the drivers name.

RED (parent: the plugin's production config from e1fb504, this commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/probe-form-oracle.property.test.js
  3 of 9 fail: census with a guard, single-package layout; fixed seed 2234; random
  seed. First counterexample: layout "singlePackage", a "let-guard" site, difference
  "the observed value differs from the owed value": no probe-form reports, and the
  root config has no rule for src/ at the root.

GREEN: the next commit adds the single-package glob.

Mutant: the parent config (no src/ glob) is the hostile mutant for this law; the
same three tests fail.

Limits: the gap-report oracle (packages/oracles/oracle-gaps/
gap-report-probe-form-oracle.property.test.mjs) shares the grammar and does not
read the layout, so its own declared reach does not include the single-package
layout. Recorded here for the owner.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): production config covers the single-packa…
079bd06
…ge layout

The production config's files glob was packages/*/src/**, so a single-package
repository with src/ at its root got no probe-form reports. The config now
names both layouts: packages/*/src and src, with the same source extensions.

Law: the probe form is enforced in production source in either layout (see the
previous commit's oracle; authority: the probe-form law's maintainer decisions of
2026-11-08 and P8 of the frozen v20 design grammar).

RED (parent: the production glob from e1fb504, the previous commit's oracle):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/probe-form-oracle.property.test.js
  3 of 9 fail (census with a guard, single-package layout; fixed seed 2234;
  random seed), difference "the observed value differs from the owed value".

GREEN (this commit): the same command, 9 of 9 pass, with the single-package
layout reached (generator, census, and constructed history). Full plugin suite:
14 files, 4291 tests pass.

Mutant: the parent glob (without src) fails the same three tests; the
layout-specific reports are the states the mutant loses.

Limits: the oracle draws `.ts` file names only, so an extension missing from the
glob (for example .tsx under src/) is not caught by this oracle. Recorded as a
gap for the owner. The gap-report probe-form oracle shares the grammar and does
not read the layout.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): a disable directive naming an oracle rul…
faf49da
…e is reported

P8 of the frozen v20 design grammar says lint rules cannot be disabled inline.
The noInlineConfig setting covers oracle files, but the production config cannot
take it without ignoring every other rule's directives in production source. The
maintainer chose to report the directive instead. This oracle owns the new rule
oracle/no-oracle-disable, which the next commit adds to both configs.

Law: a disable directive that names an oracle/* rule is reported on its own line,
in an oracle file and in production source, in the line, next-line, and block
forms (block and block-line), with or without a description, and whether the list
names only oracle rules, only other rules, or both. A directive that names no
oracle rule is not reported. A throw is a violation: the rule never throws.
Authority: P8 (frozen v20 design grammar) and the maintainer's decision on the
production-config limit (option b).

Limits written in the oracle: a blanket directive with no rule list is not seen
(ESLint suppresses the report itself), and a directive naming the rule itself can
hide the report on the line below it. Neither is generated.

Grammar: five directive forms, five rule-list kinds, a description flag, and two
locations (oracle file, production source). Constructed histories cover every form,
list kind, location, the described form, and the silent case.

Reach: each state is entered by a constructed history, and the generator (60 runs,
fixed seed 20261110) draws the rest.

RED (current tip, before the rule exists; the rule is not in either config):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/no-oracle-disable-oracle.property.test.js
  3 of 5 fail (fixed seed 20261110, random seed, calibration "a report dropped").
  First counterexample: {"location":"oracleFile","sites":[{"form":"nextLine",
  "list":"oracleOne"}]}, difference "the observed value differs from the owed
  value": the owed report is missing. The calibration fails because no clean
  history has a report yet.

GREEN: the next commit adds the rule and enables it in both configs.

Mutant: the next commit reports line and next-line forms only when the next-line
or line form is ignored; the same tests fail.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
feat(eslint-plugin-oracles): report a disable directive that names an…
65e1c85
… oracle rule

Oracle rules cannot be disabled inline (P8, frozen v20 design grammar). The
recommended config's noInlineConfig covers oracle files only. The production
config cannot take it without ignoring every other rule's directives in
production source, so the maintainer chose to report the directive instead.

The new rule oracle/no-oracle-disable reports each comment whose directive lists
an oracle/* rule: the eslint-disable, eslint-disable-line, and
eslint-disable-next-line forms, as line or block comments, with or without a
description, and whether the list is oracle only, other only, or mixed. The
report sits on the comment's own line. It is enabled in both recommended and
production. The rule does not throw: it reads comment text only.

Law: as in the oracle commit before this one (no-oracle-disable-oracle.property.test.js).
Authority: P8 and the maintainer's decision on the production-config limit (option b).

Limits: a blanket directive with no rule list is not seen, because ESLint
suppresses the report itself on the lines it covers. A directive that names
oracle/no-oracle-disable can hide the report on the line below it (its own
report is still made unless suppressed by a further directive). Neither form is
generated.

Reach: the oracle's constructed histories and generator (60 runs, fixed seed 20261110)
enter each form, each list kind, the described form, the silent case, both
locations, and the reported state.

RED (previous commit, on the tip before this rule):
  cd packages/eslint-plugin-oracles && npx vitest --run tests/no-oracle-disable-oracle.property.test.js
  3 of 5 fail (fixed seed 20261110, random seed, calibration "a report dropped").
  First counterexample: {"location":"oracleFile","sites":[{"form":"nextLine",
  "list":"oracleOne"}]}, difference "the observed value differs from the owed value".

GREEN (this commit): the same command, 5 of 5 pass. Full plugin suite: 15 files,
4296 tests pass. pnpm typecheck and pnpm format:check pass. pnpm lint:check passes
with no new warnings (the 2 warnings in packages/oracles are unchanged).

Mutant (a): the rule reads Block comments only, so line comments are missed.
  Same command: 2 of 5 fail (first counterexample a nextLine form).
Mutant (b): the directive pattern drops the next-line keyword, so next-line forms
  are missed. Same command: 2 of 5 fail.
Both mutants were run on this rule and the rule was restored.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): no-oracle-disable oracle generates quote…
10d7c1e
…d, config, self-naming, and non-directive comments

The oracle now generates the directive spellings that the rule must report
and the comments it must not: quoted rule names, inline `eslint <rule>: "off"`
config comments, a list that names oracle/no-oracle-disable itself, a
description that names an oracle rule, a non-directive comment, JSDoc, and a
string holding directive text. Its driver charges only errors that ESLint
tags with a rule id to the rule; harness errors propagate.

Law: an ESLint directive that disables an oracle/* rule is reported on the line
it begins, however spelled; a comment that ESLint does not read as a directive
is not reported.
Authority: P8 of the frozen v20 design grammar (lint rules cannot be disabled
inline; maintainer chose option b, a plugin rule that reports the directive).
P10: a harness failure is never charged to the rule. The directive grammar is
ESLint's.

Reach: the constructed histories declare each form and list kind, and the
counterexamples below are drawn from them. Per-state hit counts were not
measured in this commit.

RED (unchanged production, HEAD 65e1c85):
`npx vitest --run tests/no-oracle-disable-oracle.property.test.js` in
packages/eslint-plugin-oracles: 2 failed, 3 passed. Shrunk counterexamples:
a `config` form (finding 2) and a double-quoted and a single-quoted
`oracle/probe-form` directive (finding 1). Per-finding probe on the same
production: quoted, config, and block/line self-naming directives are not
reported, and they silence oracle/probe-form. A bare directive is reported.

GREEN: not applicable. This commit changes no production code. The oracle
stays RED until the production rule is repaired.

Mutant: a substring mutant (`comment.value.includes('oracle/')`) fails the
oracle on a `jsdoc` history, where HEAD production is silent. The mutant was
applied in place and restored; no production file is in this commit. The old
oracle passed this mutant 5/5 per the review.

Limits: blanket directives (no rule list) are not generated. The report
location for a self-naming directive is open: ESLint hides a report at or
after the directive's start, so the rule's choice of line is a maintainer
decision. Four other drivers (oracle-property-only, probe-form, fixed-seed,
replay-observation) still charge all errors to the rule. The model's
`isOracleRule` predicate is still a copy of production's and needs an
independent judgment.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): no-oracle-disable oracle covers blanket,…
478d9e7
… first-token, covered, and config forms

Extends the no-oracle-disable oracle to the review's remaining directive
law: a blanket directive with no rule list is reported; a directive that
names this rule, or has no list, sits at the file's first token; a
directive under an earlier block directive in production source sits at
the first token; an inline config comment that names an oracle rule is
reported. The model names the plugin's rules itself instead of testing a
prefix, and it generates a header comment before the first token.

Law: every ESLint directive that would silence an oracle/* rule is
reported once; the report sits on the directive's line, or at the first
token when the directive names this rule, has no list, or is covered by
an earlier block directive in production source.
Authority: P8 of the v20 design grammar (option b); maintainer decision
that blanket directives are reported wherever ESLint lets the report
through, and that a self-naming directive is reported at the first token.

Reach: the constructed histories include blanket line, next-line and
block, a directive covered by a self-naming block, a directive covered by
a blanket block, a directive under a block that names other rules, a
config comment, and a header comment. Per-state counts not measured.

RED (unchanged production, HEAD 10d7c1e):
`npx vitest --run tests/no-oracle-disable-oracle.property.test.js` in
packages/eslint-plugin-oracles: 2 failed, 3 passed. Shrunk counterexamples:
a config-comment history (F2), and a block-line history with single-quoted
names.

GREEN: not applicable here; the production repair follows in its own commit.

Limits: a directive before the first token is not generated; enable
directives are not generated.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): no-oracle-disable coverage applies in or…
311ddad
…acle files too

The rule cannot tell whether an oracle file's noInlineConfig is set, so an
earlier block directive covers a later report in both locations; the model
now says so. Law unchanged otherwise.

RED (unchanged production, HEAD 478d9e7): oracle, 2 failed / 3 passed. A
block-covered report in an oracle file is expected at the first token and
the rule places it on its own line.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): no-oracle-disable first-token reports si…
cdfdd1e
…t at the file's start under either parser

The Program node is the report location for a self-naming or blanket
directive, but its start depends on the parser: typescript-eslint starts
it at a header comment, espree at the first statement. The oracle now
generates the parser (typescript or espree) and a header comment, and
places the report at line 1, column 0: the file's start, which precedes
every directive under either parser.

Law: a report that cannot sit on its directive's line sits at the file's
start, line 1 column 0. Authority: maintainer decision (F8, option 2).

Limit recorded: a config comment that names this rule turns it off for
the file, so ESLint runs no rule and none reports it. The grammar does not
write that comment; the rule cannot close it, and the README and changeset
record the residual.

RED (unchanged production, HEAD 311ddad): oracle, 2 failed / 3 passed.
Counterexample: an oracle file parsed by espree with a header comment; the
rule reports line 2 where the law places the report at line 1.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): no-oracle-disable reports every directive…
faa73ad
… that would silence an oracle rule

The rule now reads the directives and inline config comments through
sourceCode.getDisableDirectives() and sourceCode.getInlineConfigNodes()
instead of a hand-written directive regex.

- F1: quoted rule names (bare, single, double) are the rule's names.
- F2: an inline `eslint oracle/x: "off"` config comment is reported.
- F3: a directive that names this rule, or has no list, is reported at the
  file's start (line 1, column 0): it hides its own report otherwise. A
  directive or config comment under an earlier block directive is reported
  at the file's start as well.
- F10: a blanket directive with no rule list is reported, in every form.
  A file-wide block directive is reported too, when code precedes it.
- F8 (rule side): the report is at an explicit file-start location, so it
  does not depend on whether typescript-eslint or espree gives the Program
  node its start.

Law: every ESLint directive that would silence an oracle/* rule is reported
once; the report sits on the directive's line, or at the file's start when
the directive names this rule, has no list, or lies under an earlier block
directive. Authority: P8 (option b), maintainer decisions on blanket and
self-naming directives and on the file-start location.

Reach: the constructed histories cover blanket line, next-line and block,
self-naming block and line, covered by a self-naming block, covered by a
blanket block, under a block naming other rules, config, header comment,
espree and typescript parsers. Per-state counts not measured.

RED (oracle at cdfdd1e, rule at 65e1c85, unchanged by the oracle commits): oracle
2 failed / 3 passed before this commit (blanket, covered and parser
histories). Probe on the unchanged rule: quoted, config, and self-naming
directives are not reported and silence oracle/probe-form.

GREEN (this commit): oracle 5/5 passed. Plugin suite: 15 files, 4296 tests
passed. tsc, eslint, prettier clean on the changed files.

Mutant (each applied to the rule, run through the oracle, restored):
- quotes not stripped: killed (2 failed)
- no covered relocation: killed (2 failed)
- self-naming not relocated: killed (2 failed)
- blanket not reported: killed (2 failed)
- description substring: killed (2 failed)
- config ignored: killed (2 failed)
- Program-node location: killed (2 failed)

Limits: a self-naming or blanket directive on line 1 covers the file's start,
so ESLint hides its report and no location exists. A production config comment
that turns this rule off silences it for the file; no rule can report that.
Directives that cover later lines through enable are not generated.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
refactor(eslint-plugin-oracles): oracle-property-only drops redundant…
08b6887
… importOf and report copies

- S-1: importedName already yields undefined for default and namespace
  specifiers, so the ImportSpecifier test is removed.
- S-2: rawCampaignOf returns the report's messageId and data, and the
  handler spreads it instead of copying field by field.
- S-5: malformedList has no placeholders, so it takes no data object.

Behavior-preserving: same reports, same messageIds, same data.
GREEN: plugin suite 15 files, 4296 tests passed.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): oracle-property-only oracle covers espre…
5cf1450
…e header location and non-array module settings

Extends the never-throws and location grammar:
- parser (typescript or espree) and a leading header comment, so the
  Program node's start differs between parsers (F8): a report at line 1
  must sit at the file's start under either;
- implementationModules as a non-array value, which the rule must report
  and never throw on (F6); a non-array value is read as an empty list, so
  the implementation exemption is lost.

Narrows the F9 law: a refused entry names another file or an implementation
module, and no legal history lets it exempt the file under test, so the
"exempts nothing" half is not observable and is said so in Limits.

Law: the rule never throws on a program that parses; a configuration that
it cannot judge is reported at line 1 of the file, whatever the parser.
Authority: P8, and the maintainer's option 2 for the report location (F8).

RED (unchanged production, HEAD 08b6887): oracle 2 failed / 3 passed.
- espree with a header comment: the rule reports line 2 where the law places
  the report at line 1.
- implementationModules as a string: the rule throws a TypeError
  (modules.some is not a function), which the oracle's check counts as a
  violation.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
test(eslint-plugin-oracles): a non-array implementationModules is rea…
1e548e9
…d as an empty list

The model judges an implementationModule entry as refused only while that
list is an array. A non-array value is read as an empty list (the law
written with the rule), so the entry is judged instead and is not refused.

RED: none (model correction); the implementation change follows in the next
commit. Plugin suite on this model alone: one failed (the history above).

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): oracle-property-only reports non-array mo…
aa88bd9
…dule settings and places its reports at the file start

- F6: a non-array settings.oracles.implementationModules (or
  replayWitnessModules) is read as an empty list and reported at the file
  start as malformedSetting. The rule no longer throws on it.
- F8: the configuration and stale/without-reason reports sit at an explicit
  file-start location (line 1, column 0), not the Program node, whose start
  depends on the parser (typescript-eslint starts it at a leading comment,
  espree at the first statement).

Law: the rule never throws on a program that parses; a configuration it
cannot judge is reported at line 1 of the file, whatever the parser.
Authority: P8, and the maintainer's option 2 for the location (F8).

RED (oracle at 5cf1450 + model fix 75b9201-parent, rule at HEAD before this
commit): oracle 2 failed / 3 passed: espree with a header comment reports
line 2 where the law places line 1; a string implementationModules throws.

GREEN: oracle 5/5 passed; plugin suite 15 files, 4296 tests passed; tsc and
eslint clean.

Mutant (applied, oracle run, restored): program-node location: killed (2
failed); malformed setting silent: killed (2 failed); stale at program node:
killed (1 failed); setting still throws (no normalization): killed (2 failed).

Not done in this commit: no-disabled and replayable-seed read the same
settings and now normalize them (no throw), but do not report the
malformedSetting yet. Their oracles do not generate a non-array setting,
so that report is unproved and stays open.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
fix(eslint-plugin-oracles): probe-form calibrations skip a trace the …
528ffe6
…rule threw on

The two report calibrations read observed.value as the rule's report list.
When the rule throws, the observation is { threw } and has no length, so the
calibration called .map on an object and the harness raised a TypeError
instead of the law's violation (M1, verified in this run).

RED: a probe-form mutant that throws in Program:exit gives
"TypeError: o.value.map is not a function" and "o.value.slice is not a
function" from the calibrations (7 failed).

GREEN: the same mutant now fails only on the never-throws law (no TypeError,
7 failed); the clean run passes 9/9.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
docs(eslint-plugin-oracles): document the no-oracle-disable residuals…
87b1dc5
… and the file-start location

- README and changeset describe every directive the rule reports: quoted
  names, blanket directives, inline config comments naming an oracle rule.
- The report location: the directive's line, or line 1 column 0 when the
  directive names this rule, has no list, or is under an earlier block
  directive (F3, F8).
- Residuals the rule cannot close: a directive on line 1 that names this
  rule or has no list covers the file's start; a production config comment
  that turns this rule off silences it for the file (F10).
- noInlineConfig in oracle files ignores every rule's directives, not only
  the oracle rules, and the changeset says so as a behavior change (F5).
- A non-array implementationModules or replayWitnessModules is reported at
  line 1 as malformedSetting (F6).
- The probe-form production globs move into their own paragraph (S-11), and
  the long README paragraph is rewrapped (S-13).

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
refactor(eslint-plugin-oracles): oracle-property-only shares the raw-…
d231a8c
…campaign message tail (S-3)

rawCampaign and rawProperty differ only in their first clause, so the shared
advice is one constant. Message ids and the rendered text are unchanged.

GREEN: plugin suite 15 files, 4296 tests passed; prettier clean.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
refactor(eslint-plugin-oracles): one lint charge helper for the five …
dfd6f03
…oracle drivers (F7, S-14, S-15)

Each driver wrapped its lint run in a try that recorded every error as the
rule's throw. Only an error that ESLint tags with a rule id is the rule's
throw; any other error is a harness failure, which must propagate (P10). The
shared helper tests/lint-observation.js returns { messages } or { threw },
charging only the tagged error. The five drivers use it:
probe-form, fixed-seed, replay-observation, oracle-property-only, and
no-oracle-disable (which already had the same logic inline).

Authority: P10 and decision 13 ("a harness failure stops the run with one
diagnostic and is never charged").

RED (HEAD driver, probe-form test, parser replaced by an invalid value): the
harness error is recorded as "threw: Key parser: Expected object ..." and the
rule is blamed (7 failed, each a census violation).
GREEN (new driver, same injection): the TypeError propagates and the run stops
(9 failed, all with the harness error).

Limit: the threw text changes from error.message to String(error), so the
observation carries the error's name too. The suite is green on this change
(15 files, 4296 tests).

Co-authored-by: Isaac <[email protected]>
@KyleAMathews Isaac
refactor(eslint-plugin-oracles): import no-oracle-disable in rule ord…
16ce17b
…er (S-8)

Module evaluation has no side effects, so the order is cosmetic. The import
now sits in the same order as the rules object.

GREEN: plugin suite 15 files, 4296 tests passed.

Co-authored-by: Isaac <[email protected]>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f6d38a39-68fd-4693-9f9c-4562cd41e632
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

  • X
  • Mastodon
  • Reddit
  • LinkedIn
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@changeset-bot

changeset-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 16ce17b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 1 package
Name Type
@tanstack/eslint-plugin-oracles Patch

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pkg-pr-new

pkg-pr-new Bot commented Nov 10, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@6
npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@6

commit: 16ce17b

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@KyleAMathews
KyleAMathews merged commit f4d5a0e into main Nov 10, 2026
8 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@KyleAMathews
KyleAMathews deleted the fix/lint-never-throw branch November 10, 2026 20:00
KyleAMathews added a commit that referenced this pull request Nov 10, 2026
@KyleAMathews Isaac
docs(oracles): follow #6 in setup, and require a fixed seed
f5011b0
#6 made `recommended` turn off inline directives and widened the
`production` glob to `src/**`, so setup now adds both presets as they are,
without the workaround blocks. The plugin README lists no-oracle-disable
and the new glob. The design skill now asks for at least one fixed seed
(an empty list will be refused) and treats a random-only kill as a gap
until its history is kept.

Co-authored-by: Isaac <[email protected]>
@KyleAMathews KyleAMathews mentioned this pull request Nov 11, 2026
fix(eslint-plugin-oracles): load and parse with the consumer's parser #13
Merged
KyleAMathews added a commit that referenced this pull request Nov 11, 2026
@KyleAMathews Isaac
fix(eslint-plugin-oracles): load and parse with the consumer's parser (…
745502b
…#13)

* fix(eslint-plugin-oracles): load and parse with the consumer's parser

`oracle/no-unit-test` imported `@typescript-eslint/parser` at module load.
That parser refuses TypeScript 7, so importing the plugin, and every one of
its configs, threw in a repository that pins TypeScript 7.0.2 (field report
round 3, nanostores). It also threw when `typescript` was absent.

The rule now parses each imported test module with the parser and parser
options that ESLint uses for the file it lints, through ESLint's own Linter,
so any configured parser works, espree included. The parse cache is keyed on
those language options too. typescript-eslint's type-aware options are left
out of that parse: the rule reads syntax and scope only, and a nested
`project` parse changed the program that type-aware rules then read (single-
run mode). `@typescript-eslint/parser` moves to devDependencies.

Also fixed, found by the extended oracle: under espree the rule dropped every
reference (espree scopes have no `isValueReference`) and reported reaching
tests as undeclared. And a module that did not parse was skipped silently:
when a test reaches no part, each import of a module that the file's parser
cannot read is now reported (unparsedModule) with the parser's error. The rule
never throws on it.

Evidence

- Law: the plugin loads whatever the consumer's TypeScript (field report);
  rules never throw (maintainer decision 2026-11-08, #6); an imported module
  is parsed as ESLint parses the linted file (consumer-parser task). The
  narrowing of the report to files with an unreached test, and the strip of
  type-aware options, need maintainer confirmation.
- Reach: new lint.part-reach.parsers family gates espree-oracle,
  espree-oracle-test-uses-part, oracle-imports-type-syntax-under-espree
  (fixed seed 82, 24, 56 of 103); unparseable-companion gates
  oracle-imports-unparseable-companion (20); new parser-switch law gates the
  type-syntax state (22). The TS environment is a focused test
  (consumer-typescript.test.js, 3 child-process cases).
- RED (d1b508a rule): 13 of 84 fail. TS 7 stub: "typescript-eslint does not
  support TS 7.0."; absent: "Cannot find module 'typescript'"; espree:
  "unit:t0" read as "undeclared:t0"; broken import: "unparsed:broken.ts"
  missing.
- GREEN: plugin suite 17 files, 4310 tests; typecheck, eslint, prettier,
  sherif, publint clean. A copy of the nanostores repro loads the plugin and
  lints an espree-readable oracle file.
- Mutant: top-level parser import (consumer-typescript), fixed espree or
  fixed typescript-eslint (parsers, parser-switch), a throwing parse error
  (charged as threw:), the old isValueReference test (parsers), a cache that
  ignores the parser (parser-switch), kept type-aware options
  (typed-lint-isolation), eager reports (parsers, unparseable-companion).
- Limits: the linted file's options, not the imported module's own, so a JS
  oracle importing a TS module with type syntax gets the report when a test
  is unreached; transitive unparseable modules are not reported; two parsers;
  a stub of TS 7's shape; ESLint 10 not run. Under TS 7, typescript-eslint
  itself cannot load, so .ts oracle files still cannot be linted there.
  no-unit-test.js grows 269 bytes min+gzip; the install drops the
  @typescript-eslint/parser tree.

Co-authored-by: Isaac <[email protected]>

* fix(eslint-plugin-oracles): report only parse failures as unparsed; ignore inline config in imported modules

Review round on the consumer-parser change:
- Only reading and parsing an imported module is caught. An error in the
  rule's own analysis of a parsed module is a rule bug and is no longer
  reported as a parse failure.
- The nested Linter ignores the imported module's eslint comments
  (noInlineConfig), so they run no core rules and print no env warnings.
- The changeset states the two behaviors to expect: a module is parsed with
  the linted file's parser, and only direct imports report a parse failure.

Co-authored-by: Isaac <[email protected]>

---------

Co-authored-by: Isaac <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@KyleAMathews
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.