Workflow · last verified 2026-07-26

Validating the validator: fixture the first check before you trust it

The job this workflow does

Some files carry more authority than their contents deserve. A coordination ledger, a service registry, a schema snapshot, a config that eight things read on boot — nobody re-derives what is in them, because re-deriving is the cost the file exists to remove. That is exactly what makes them worth checking, and it is also what makes the check dangerous: the moment a check runs against an artifact everyone trusts, its verdict inherits that trust. People act on the output without re-reading the input, which is the whole point, and which is why a wrong verdict propagates at the speed of a right one.

The failure has a specific and easily-missed shape. A checker bug does not surface as "the checker is broken." It surfaces as a plausible, specific, actionable defect report about the artifact — with a line number, a count, and a remediation. It reads exactly like a real find, because a real find looks the same. Nothing about the report distinguishes the two, and the only thing that can is a fixture: a case where you already know the answer.

This is the AI-native version of a very old problem, and agents sharpen it. A capable agent asked to "check this file" will write a check, run it, report findings, and file tickets in one pass, with the fluency of something that verified its work. It did verify — against its own reading of the format, which is the thing in question. The speed is real and worth having; the missing step is one fixture between the check and the ticket.

Three cases in one week

These are consecutive, unplanned, and the same shape each time: the artifact everyone trusted had never been checked, and the first thing to check it had never been checked either.

1. The check filed its own bug as a defect in the file

An ad-hoc script was written to validate the rows of this repo's coordination ledger — a pipe-delimited Markdown table with a fixed eight-column shape. It split each row on every | it found. Two rows contain a regular expression in their text, written with the standard GFM escape (\|) so that the pipe renders as a literal instead of a cell boundary. The escape is correct, the rows render correctly, and they parse as exactly eight cells. The naive split saw nine fields and reported them broken.

The report was specific and confident, so it was believed. It was filed as a defect, prioritised, and queued as a content candidate about the coordination spine nobody validates — a claim that would have been published on a site whose entire thesis is as-built proof. It was killed at the last gate before authoring, on the discovery that the two rows had never been broken at all. The decision record retiring it is blunt about the mechanism: the validator "split on every pipe and reported its own bug as a ledger defect."

2. The real validator reproduced the identical bug

The permanent fix was a committed structural gate for the same ledger, run as part of a weekly lint. Its first pass split on every pipe too — the same defect, independently, in the code written specifically to prevent it. It was caught when its author stopped reasoning about the format and read the raw bytes of the two rows in question.

The whole correction is one lookbehind, and it is worth seeing how small the difference between a working gate and a gate that manufactures defects actually is:

// scripts/lint-monday-artifacts.mjs - the delimiter split
// WRONG: treats the GFM escape \| as a cell boundary
const cells = line.split('|').slice(1, -1);

// RIGHT: a backslash-escaped pipe is content, not a delimiter
const cells = line.split(/(?<!\\)\|/).slice(1, -1);

The comment above that line in the shipped file names the incident rather than the rule, which is the version that survives a future rewrite: splitting naively on every pipe "reports a correctly-escaped row as broken, which is how [those two rows] came to be filed as defects."

3. A decision made, specced and dispatched on a premise the code contradicted

The third case is the same failure one level up. A ticket asserted that a particular degraded grading path left no server-side record that a degrade had happened — a real problem, if true, because it would silently poison the platform's item statistics. It was surfaced, escalated, decided in the decision record, written into a spec, and dispatched to an implementation desk.

The code had contradicted the premise for weeks. src/lib/grade.ts already tags that path's attempts with a 'self_graded' context at write time, on a column added by migration 0007_attempt_context.sql — so the missing record the entire chain was built on existed the whole time. It was caught only because the implementing desk read the code before writing any, and the ticket was dropped with a regression test added in place of the schema change.

The generalisation across all three: the gates in this estate check output — copy against the product record, model ids against vendor docs, content against a structural linter. Nothing checked an input's premise against the artifact it described. A checker is just the most concentrated version of that gap, because its input is the artifact.

The workflow

1. Read the raw bytes before you write the check

Not the rendered view, not your memory of the format, not a sample you generated to reason about — the actual bytes of the actual file, including at least one row or record you expect to be awkward. Every case above traces to the same omission: the format was reasoned about rather than read, and the reasoning was almost right. Escapes, encodings, BOMs, trailing separators, and empty-but-present fields are exactly the details a plausible mental model smooths over.

2. Write the known-good fixture first, and make it the awkward one

Before the failing case, write the case that must pass — and pick the input that is legal but unusual, because that is the one a naive check gets wrong. A fixture of ordinary rows proves nothing; every broken checker passes those. The shipped test for the ledger gate asserts a well-formed table that deliberately includes a GFM-escaped pipe inside a cell, which is precisely the input that killed both earlier attempts:

// test/gates.test.mjs - the known-good case IS the awkward case
it('passes a well-formed queue, including a GFM-escaped pipe inside a Title', () => {
  const ok = [
    '| ID | Opened | Source | Owner | Pri | Status | Title | Link |',
    '|---|---|---|---|---|---|---|---|',
    '| Q-001 | 2026-07-25 | ceo | eng | P2 | open | a title | link |',
    String.raw`| Q-002 | 2026-07-25 | ceo | eng | P2 | open | gated on /spend\|budget/i | src/lib/llm.ts |`,
  ].join('\n');
  expect(validateQueue(ok)).toEqual([]);
});

Write this test and both earlier versions of that validator fail immediately, in under a second, before anything is filed.

3. Write the known-bad fixture: inject the defect, assert the exact message

Now inject the defect the check exists to catch, one per structural rule, and assert the specific message rather than just a non-empty result. Asserting only "it found something" lets a check pass while finding the wrong thing for the wrong reason — which, in a checker whose failure mode is misattribution, is the failure you are trying to exclude. The same test file asserts four injected defects and matches each message individually: the unescaped pipe, a closed row left in a live file, a second closed row, and a duplicate id.

This is the step that converts a check from an opinion into a gate. A check that has never been observed to fail is indistinguishable from a check that cannot fail, and the two behave identically right up until the day you need one of them.

4. Give the check an input override so it can run against a fixture

A check hardcoded to the production artifact can only be exercised by damaging the production artifact. Take one argument. In the shipped gate it is a single flag whose comment states its entire purpose — reading a different ledger "for exercising the gate, nothing else":

// scripts/lint-monday-artifacts.mjs
const queuePath = process.argv.find((a) => a.startsWith('--queue='))?.slice(8)
  ?? join(root, 'docs/QUEUE.md');

This is also what keeps the check honest under change. Every later edit to the parser can be re-run against both fixtures at no cost, so the second bug in the checker is caught by the same test that would have caught the first.

5. Calibrate thresholds against the real corpus, and store the measurement beside the constant

Any check with a number in it — a similarity threshold, an n-gram width, a staleness budget — has a number that was either measured or asserted, and the code looks identical either way. Measure it, then write the measurement next to the constant so the next reader can tell which kind it is.

This platform's content linter gained an overlap check that flags a published demo question reproducing text from the paid item bank. The obvious question is what window counts as a copy, and the author's first answer was an unmeasured comment. Measured against the live bank, an 8-word window produced 184 collisions between items that were independently written — one house voice on one subject repeats itself — across 691 items and 96,446 shingles. Narrowing the window to 6 words nearly tripled the noise to 485; widening it to 9 only reduced it to 120. There is no width that separates the two populations cleanly, so the constant ships with that measurement recorded above it and a claim calibrated to match: a hit is strong evidence of a copy, not proof of one.

The discipline is not "measure everything." It is that a threshold with a number and no recorded measurement is an assertion wearing the costume of a result, and six months later nobody can tell.

6. Run it against the live artifact, and expect red

A new check that passes on its first real run is a finding, not a relief — it usually means the check is looking at the wrong thing, or that the artifact it examines does not exist. Three gates added here in one week each went red immediately and each red was real: the content-lint extension found 16 pre-existing flaws in published demo quizzes; the intake check found two skills that were routed work they had no step to read; and the ledger validator's own first run found that the ticket which commissioned it had the premise backwards.

Make "nothing to check" a failure rather than a pass. In this repo's freshness checks, a log file that contains only a header must read as no artifact, never as fresh — a scheduled job that reports healthy while doing nothing is the exact silence a gate exists to break, and it is the one thing a naive "did it error?" check will never see.

Failure modes

• The checker's bug arrives dressed as an artifact defect. It is specific, actionable, and wrong, and every step downstream treats it as a finding. This is the failure the whole workflow exists for — two independent instances in one week here, the second one inside the fix for the first.
• A green that means "there was nothing to look at." A missing file, an empty section, a glob that matched zero paths, a header-only log. Absence must be a failure state with its own message, or the check reports health on a job that stopped running.
• A threshold asserted rather than measured. Indistinguishable in the code from a calibrated one, and the tell is whether the measurement is written down.
• A check copy-pasted onto the wrong artifact. When checks are cloned across similar targets, the clone keeps the original's subject. The intake check here explicitly catches the case where a step names a sibling instead of itself — a defect that is invisible on read, because the text is coherent and only the name is wrong.
• Gating the output while nothing gates the premise. The mature version of this estate had five gates on what agents write and none on what they were told. That is case 3, and it is not fixed by adding a sixth output gate.
• Trusting a cited file:line. Line numbers drift with every edit above them; the ticket in case 3 cited a line that had already moved by the time it was worked. Cite the file and the symbol, verify the line at the moment you use it.

Verification

You know this worked when the fixtures exist as tests rather than as a one-time exercise, so the next edit to the checker is gated by the same cases that gated the first. On this repo the gates' fixtures live in a single test file that runs with the ordinary suite, so a regression in a checker fails the same way a regression in the product does:

# the fixtures run with everything else - no separate ritual to remember
npm test

# the gate itself, against the live artifacts; non-zero exit is the point
npm run lint:monday

The observable that matters most is the last one, and it is a lagging indicator you should actively want: the gate eventually fails on its own author. That happened here within days — a malformed row written by the same session that had commissioned the validator, caught on the next run. A gate that has never inconvenienced the person who built it has not yet been shown to be load-bearing.

When not to use this workflow

Skip the fixture pair for a check whose verdict you will re-derive by hand anyway — a one-off grep you run, read, and act on in the same minute, where you are the fixture. The cost of this workflow is real (two tests and an argument) and it buys nothing when nothing downstream consumes the verdict without looking.

The exemption is narrower than it sounds, and it expires quietly. The ad-hoc script in case 1 was written as exactly that kind of one-off, and it was believed by three subsequent steps that never saw the file. The moment a check's output becomes an input to something else — a ticket, a decision, a dashboard, another agent — it is no longer a one-off and it needs its fixtures, regardless of how it was originally intended. If you are about to file something on a checker's say-so, you have crossed the line.

Changelog

• 2026-07-26 — Initial version. All three cases are dated incidents from this platform's own build, verified against the shipped gate scripts, their fixture tests, and the decision record that retired the false premise.