Workflow · last verified 2026-07-26

Validating the validator: fixture the first check before you trust it

The job this workflow does

Some files carry more authority than their contents deserve. A coordination ledger, a service registry, a schema snapshot, a config that eight things read on boot — nobody re-derives what is in them, because re-deriving is the cost the file exists to remove. That is exactly what makes them worth checking, and it is also what makes the check dangerous: the moment a check runs against an artifact everyone trusts, its verdict inherits that trust. People act on the output without re-reading the input, which is the whole point, and which is why a wrong verdict propagates at the speed of a right one.

The failure has a specific and easily-missed shape. A checker bug does not surface as "the checker is broken." It surfaces as a plausible, specific, actionable defect report about the artifact — with a line number, a count, and a remediation. It reads exactly like a real find, because a real find looks the same. Nothing about the report distinguishes the two, and the only thing that can is a fixture: a case where you already know the answer.

This is the AI-native version of a very old problem, and agents sharpen it. A capable agent asked to "check this file" will write a check, run it, report findings, and file tickets in one pass, with the fluency of something that verified its work. It did verify — against its own reading of the format, which is the thing in question. The speed is real and worth having; the missing step is one fixture between the check and the ticket.

Three cases in one week

These are consecutive, unplanned, and the same shape each time: the artifact everyone trusted had never been checked, and the first thing to check it had never been checked either.

1. The check filed its own bug as a defect in the file

An ad-hoc script was written to validate the rows of this repo's coordination ledger — a pipe-delimited Markdown table with a fixed eight-column shape. It split each row on every | it found. Two rows contain a regular expression in their text, written with the standard GFM escape (\|) so that the pipe renders as a literal instead of a cell boundary. The escape is correct, the rows render correctly, and they parse as exactly eight cells. The naive split saw nine fields and reported them broken.

The report was specific and confident, so it was believed. It was filed as a defect, prioritised, and queued as a content candidate about the coordination spine nobody validates — a claim that would have been published on a site whose entire thesis is as-built proof. It was killed at the last gate before authoring, on the discovery that the two rows had never been broken at all. The decision record retiring it is blunt about the mechanism: the validator "split on every pipe and reported its own bug as a ledger defect."

2. The real validator reproduced the identical bug

The permanent fix was a committed structural gate for the same ledger, run as part of a weekly lint. Its first pass split on every pipe too — the same defect, independently, in the code written specifically to prevent it. It was caught when its author stopped reasoning about the format and read the raw bytes of the two rows in question.

The whole correction is one lookbehind, and it is worth seeing how small the difference between a working gate and a gate that manufactures defects actually is:

// scripts/lint-monday-artifacts.mjs - the delimiter split
// WRONG: treats the GFM escape \| as a cell boundary
const cells = line.split('|').slice(1, -1);

// RIGHT: a backslash-escaped pipe is content, not a delimiter
const cells = line.split(/(?<!\\)\|/).slice(1, -1);

The comment above that line in the shipped file names the incident rather than the rule, which is the version that survives a future rewrite: splitting naively on every pipe "reports a correctly-escaped row as broken, which is how [those two rows] came to be filed as defects."

3. A decision made, specced and dispatched on a premise the code contradicted

The third case is the same failure one level up. A ticket asserted that a particular degraded grading path left no server-side record that a degrade had happened — a real problem, if true, because it would silently poison the platform's item statistics. It was surfaced, escalated, decided in the decision record, written into a spec, and dispatched to an implementation desk.

The code had contradicted the premise for weeks. src/lib/grade.ts already tags that path's attempts with a 'self_graded' context at write time, on a column added by migration 0007_attempt_context.sql — so the missing record the entire chain was built on existed the whole time. It was caught only because the implementing desk read the code before writing any, and the ticket was dropped with a regression test added in place of the schema change.

The generalisation across all three: the gates in this estate check output — copy against the product record, model ids against vendor docs, content against a structural linter. Nothing checked an input's premise against the artifact it described. A checker is just the most concentrated version of that gap, because its input is the artifact.

The other direction: three greens that were not evidence

Cases 1 and 2 are false reds, and a false red is the loud failure — somebody chases it. The opposite is cheaper to ship and much harder to notice, because nothing asks you to look: a check, a measurement, or a data source that reports success it never earned. Three from the same fortnight here, and two of them were nested inside each other in a single ticket.

A guard for absent is not a guard for emptied

One of this repo's weekly gates asserts that every skill routed coordination work has a step to read that work. To know which skills those are, it derives the list from another script's source with a regular expression rather than restating it — a deliberate choice, because a hardcoded copy rots exactly the way the defect it guards did. The author saw the obvious hazard and guarded it: if the list cannot be found at all, the gate fails with a message saying it is "running blind." The comment beside it calls a silent stop "the exact failure mode it exists to catch."

Then the list was refactored. It became a composition of per-desk arrays via spread syntax — a tidier shape, and one that kept the declaration the regex was matching perfectly intact. The pattern still matched. The names it harvested no longer lived inside the matched block, so the routed set fell from 21 names to 15, six skills stopped being checked at all, and the gate printed ok. It was caught by a desk reading the gate's output against what it expected, not by the gate.

The mechanism deserves its own sentence, because it is not a bug in the regex: the guard covered the list being missing, and nothing covered it being emptied. A check derived from a shape is coupled to that shape, and the dangerous refactor is the one that preserves the shape while changing the content. The fix is not a better pattern, it is a second assertion — that what was extracted is non-empty, per group, with a message naming the consequence rather than the symptom:

// scripts/lint-monday-artifacts.mjs - the assertion that was missing
for (const [, desk, list] of perDesk) {
  if (!quoted(list).length) problems.push(
    `huddle.js: DESK_SKILLS.${desk} is empty — that desk's queue rows are unowned`);
}

The exit status that came from the wrong command

The nested pair starts with a ticket asserting that a tool reported success while failing — the kind of claim that sends someone looking for a bug in the tool. The tool was fine. It exits non-zero on that failure, reliably, twice out of two runs. The false green was in how the exit status had been measured: the command was run through a pipe to keep the output readable, and a shell reports a pipeline's status as the status of its last command. The reading came from the pager, which succeeded. The tool's failure was never in the number being read.

This is worth internalising as a habit rather than a fact, because the pipe is added for an innocent reason — you wanted to see less output — and it silently changes what you are measuring. Measure the command you care about without a pipe, or ask for the right element explicitly:

# WRONG: $? is the pager's status, so a failing command reads as success
some-tool analyze | head -40
echo $?

# RIGHT (bash): either take the pipeline's leftmost failure...
set -o pipefail
some-tool analyze | head -40
echo $?

# ...or read the element you actually mean
some-tool analyze | head -40
echo "${PIPESTATUS[0]}"

PIPESTATUS has to be read by the very next command: it is repopulated by each command that runs, so the common habit of capturing s=$? first leaves ${PIPESTATUS[0]} reporting the assignment's status instead — zero, always. Both this and pipefail are bash features rather than POSIX sh, so under a /bin/sh shebang neither exists and the honest fix is to stop piping the thing you are measuring.

The data source that still answered

Underneath both of those sat the defect neither was looking at, and it is the most expensive of the three. The tool in question maintains a local index that sessions query for callers and blast radius. Its on-disk schema no longer matched the version the current build expected, so every run forced a full rebuild, and every full rebuild died partway through inside a vendored storage layer. That is the loud part, and it is not the problem.

The problem is that the old index kept answering. Queries returned symbols, at plausible paths, with plausible line numbers — a partial merge left behind by a crashed incremental run. Nothing about a read looked wrong, and every read was silently incomplete. A "no callers found" from an index in that state is indistinguishable from a genuine "no callers found", and the second one is a green light to change the function.

The prescription is the same as the rest of this page, applied to a data source rather than a checker: give it a fixture whose answer you already know. What runs here now is a verification script that samples a symbol from the newest committed source file and asserts the index resolves it at the right file and line, alongside checks for the two on-disk signatures of a crashed rebuild. Non-zero means every navigation answer in that session is a false green and grep is the fallback. It is a small script, and it converted "the index feels stale" into a fact a session can act on before it acts on the index.

The control that would have deployed, reported healthy, and done nothing

The three above are checkers and a data source. The fourth is a live control in front of real users, and it is the worst of the set, because a control has a deploy date and nobody re-reads it afterwards. This platform's coach classifies each learner's own turns before the tutor sees them, to catch a message trying to override the assistant's configuration. It is deliberately fail-open: offline, timeout, network error, an upstream refusal, an unparseable verdict, a tripped circuit breaker — every one of those paths returns the same unavailable status and the turn proceeds. That is the right trade for a tutoring product, where a classifier outage must not take the tutor down with it. It also means a completely dead control and a completely healthy one produce the same observable behaviour: none.

Two ordinary facts met. The classifier runs on a different model from everything else here — a small open-weights safeguard model, not the tier the tutor uses — so its model identifier appears in exactly one place. And production routes model calls through an authenticated vendor gateway, while the offline and live evaluation suites call the provider directly, so the gateway hop is taken only on the deployed path. That identifier had therefore never once traversed the gateway that would carry it in production. Had the gateway rejected it, the request would have returned an HTTP error, the judge's outer catch would have converted it to unavailable, and every turn would have proceeded exactly as it does now: no exception surfaced, no failed request in the application, no visible difference in a single coach reply. The security layer would have been inert from the first deploy, and nothing in the system was shaped to say so.

The pre-release review that spotted the gap could not close it — verifying a model pin against vendor documentation is a different job from putting a byte through the wire, and that reviewer makes no live calls by design. What closed it was one call through the real deployed path, and the design of that call is the part to copy: it asserted a positive classification, not the absence of an error. Asserting that the request did not throw passes against a dead control, because a dead control is precisely the thing that does not throw. Only an input the classifier is required to flag, coming back flagged, separates a working control from a silent one. The honest state afterwards is worth recording as well, since it is the position most teams are actually in: that coverage is one manual check, not a standing test. The evaluation suite that runs on every change stubs the classifier to unavailable by construction, so that it does not pay for a second model on every pedagogy case — which makes it structurally incapable of ever catching this.

The workflow

1. Read the raw bytes before you write the check

Not the rendered view, not your memory of the format, not a sample you generated to reason about — the actual bytes of the actual file, including at least one row or record you expect to be awkward. Every case above traces to the same omission: the format was reasoned about rather than read, and the reasoning was almost right. Escapes, encodings, BOMs, trailing separators, and empty-but-present fields are exactly the details a plausible mental model smooths over.

2. Write the known-good fixture first, and make it the awkward one

Before the failing case, write the case that must pass — and pick the input that is legal but unusual, because that is the one a naive check gets wrong. A fixture of ordinary rows proves nothing; every broken checker passes those. The shipped test for the ledger gate asserts a well-formed table that deliberately includes a GFM-escaped pipe inside a cell, which is precisely the input that killed both earlier attempts:

// test/gates.test.mjs - the known-good case IS the awkward case
it('passes a well-formed queue, including a GFM-escaped pipe inside a Title', () => {
  const ok = [
    '| ID | Opened | Source | Owner | Pri | Status | Title | Link |',
    '|---|---|---|---|---|---|---|---|',
    '| Q-001 | 2026-07-25 | ceo | eng | P2 | open | a title | link |',
    String.raw`| Q-002 | 2026-07-25 | ceo | eng | P2 | open | gated on /spend\|budget/i | src/lib/llm.ts |`,
  ].join('\n');
  expect(validateQueue(ok)).toEqual([]);
});

Write this test and both earlier versions of that validator fail immediately, in under a second, before anything is filed.

3. Write the known-bad fixture: inject the defect, assert the exact message

Now inject the defect the check exists to catch, one per structural rule, and assert the specific message rather than just a non-empty result. Asserting only "it found something" lets a check pass while finding the wrong thing for the wrong reason — which, in a checker whose failure mode is misattribution, is the failure you are trying to exclude. The same test file asserts four injected defects and matches each message individually: the unescaped pipe, a closed row left in a live file, a second closed row, and a duplicate id.

This is the step that converts a check from an opinion into a gate. A check that has never been observed to fail is indistinguishable from a check that cannot fail, and the two behave identically right up until the day you need one of them.

4. Give the check an input override so it can run against a fixture

A check hardcoded to the production artifact can only be exercised by damaging the production artifact. Take one argument. In the shipped gate it is a single flag whose comment states its entire purpose — reading a different ledger "for exercising the gate, nothing else":

// scripts/lint-monday-artifacts.mjs
const queuePath = process.argv.find((a) => a.startsWith('--queue='))?.slice(8)
  ?? join(root, 'docs/QUEUE.md');

This is also what keeps the check honest under change. Every later edit to the parser can be re-run against both fixtures at no cost, so the second bug in the checker is caught by the same test that would have caught the first.

5. Calibrate thresholds against the real corpus, and store the measurement beside the constant

Any check with a number in it — a similarity threshold, an n-gram width, a staleness budget — has a number that was either measured or asserted, and the code looks identical either way. Measure it, then write the measurement next to the constant so the next reader can tell which kind it is.

This platform's content linter gained an overlap check that flags a published demo question reproducing text from the paid item bank. The obvious question is what window counts as a copy, and the author's first answer was an unmeasured comment. Measured against the live bank, an 8-word window produced 184 collisions between items that were independently written — one house voice on one subject repeats itself — across 691 items and 96,446 shingles. Narrowing the window to 6 words nearly tripled the noise to 485; widening it to 9 only reduced it to 120. There is no width that separates the two populations cleanly, so the constant ships with that measurement recorded above it and a claim calibrated to match: a hit is strong evidence of a copy, not proof of one.

The discipline is not "measure everything." It is that a threshold with a number and no recorded measurement is an assertion wearing the costume of a result, and six months later nobody can tell.

6. Run it against the live artifact, and expect red

A new check that passes on its first real run is a finding, not a relief — it usually means the check is looking at the wrong thing, or that the artifact it examines does not exist. Three gates added here in one week each went red immediately and each red was real: the content-lint extension found 16 pre-existing flaws in published demo quizzes; the intake check found two skills that were routed work they had no step to read; and the ledger validator's own first run found that the ticket which commissioned it had the premise backwards.

Make "nothing to check" a failure rather than a pass. In this repo's freshness checks, a log file that contains only a header must read as no artifact, never as fresh — a scheduled job that reports healthy while doing nothing is the exact silence a gate exists to break, and it is the one thing a naive "did it error?" check will never see.

7. Assert the count, not just the match — and re-prove it after every refactor of what it reads

Any check that derives its subject from somewhere else — a list extracted from source, a set of files from a glob, records parsed out of a log — has two failure modes and usually guards one. It guards "I could not find it." It rarely guards "I found it and it was empty," which is the one a refactor produces. So write both, and make the count itself the assertion: n names, at least one per group, and a message that names the consequence ("that desk's rows are unowned") rather than the symptom ("list empty"). A count printed on success is worth the line it costs, because it turns the next silent shrink into something a reader notices.

The same discipline applies to a check that has been green for a while. A green earns no trust it has not been re-tested for; the refactor that empties a list is invisible in a diff of the checker, because the checker did not change. When you refactor the thing a gate reads, run the gate and read its numbers, not its exit code.

Failure modes

• The checker's bug arrives dressed as an artifact defect. It is specific, actionable, and wrong, and every step downstream treats it as a finding. This is the failure the whole workflow exists for — two independent instances in one week here, the second one inside the fix for the first.
• A green that means "there was nothing to look at." A missing file, an empty section, a glob that matched zero paths, a header-only log. Absence must be a failure state with its own message, or the check reports health on a job that stopped running.
• A threshold asserted rather than measured. Indistinguishable in the code from a calibrated one, and the tell is whether the measurement is written down.
• A check copy-pasted onto the wrong artifact. When checks are cloned across similar targets, the clone keeps the original's subject. The intake check here explicitly catches the case where a step names a sibling instead of itself — a defect that is invisible on read, because the text is coherent and only the name is wrong.
• Gating the output while nothing gates the premise. The mature version of this estate had five gates on what agents write and none on what they were told. That is case 3, and it is not fixed by adding a sixth output gate.
• A guard for the subject being absent, with nothing for it being emptied. The refactor that keeps the shape and drops the content walks past the guard and the gate prints ok. Assert the count, per group, with a message naming the consequence.
• An exit status read through a pipe. A shell reports a pipeline's status as its last command's, so piping a check into a pager makes every failure read as a pass. This one is self-inflicted at the moment you decide to shorten the output.
• A stale data source that still answers. A frozen index, a cached listing, a partially-rebuilt graph: reads succeed, look plausible, and are incomplete. "Nothing found" from a broken index is indistinguishable from "nothing exists" and reads as permission to proceed.
• A fail-open control tested for "it did not error." A control that degrades to allow returns the success-shaped answer when it is dead, so an assertion that nothing threw is satisfied by its own failure. Assert a positive detection on an input the control must catch. The risk concentrates wherever the deployed path differs from the tested one — a gateway, a proxy, an auth hop, a model identifier that only production resolves.
• Trusting a cited file:line. Line numbers drift with every edit above them; the ticket in case 3 cited a line that had already moved by the time it was worked. Cite the file and the symbol, verify the line at the moment you use it.

Verification

You know this worked when the fixtures exist as tests rather than as a one-time exercise, so the next edit to the checker is gated by the same cases that gated the first. On this repo the gates' fixtures live in a single test file that runs with the ordinary suite, so a regression in a checker fails the same way a regression in the product does:

# the fixtures run with everything else - no separate ritual to remember
npm test

# the gate itself, against the live artifacts; non-zero exit is the point
npm run lint:monday

The observable that matters most is the last one, and it is a lagging indicator you should actively want: the gate eventually fails on its own author. That happened here within days — a malformed row written by the same session that had commissioned the validator, caught on the next run. A gate that has never inconvenienced the person who built it has not yet been shown to be load-bearing.

When not to use this workflow

Skip the fixture pair for a check whose verdict you will re-derive by hand anyway — a one-off grep you run, read, and act on in the same minute, where you are the fixture. The cost of this workflow is real (two tests and an argument) and it buys nothing when nothing downstream consumes the verdict without looking.

The exemption is narrower than it sounds, and it expires quietly. The ad-hoc script in case 1 was written as exactly that kind of one-off, and it was believed by three subsequent steps that never saw the file. The moment a check's output becomes an input to something else — a ticket, a decision, a dashboard, another agent — it is no longer a one-off and it needs its fixtures, regardless of how it was originally intended. If you are about to file something on a checker's say-so, you have crossed the line.

Changelog

• 2026-07-26 — Initial version. All three cases are dated incidents from this platform's own build, verified against the shipped gate scripts, their fixture tests, and the decision record that retired the false premise.
• 2026-07-26 — Added the false-green section (a guard for absent that was not a guard for emptied, an exit status read through a pipe, and an index that answered from a partial rebuild), a seventh workflow step on asserting the count, and three failure modes. The bash exit-status forms ($? after a pipeline, set -o pipefail, PIPESTATUS[0], and the fact that capturing $? first clears PIPESTATUS) were each run and observed rather than recalled.
• 2026-08-10 — Added a fourth false green, from an incident dated 2026-07-25: a fail-open control whose model identifier had never traversed the authenticated gateway that carries it in production, so a rejection there would have left it deployed, healthy-looking and inert. Closed by a live call asserting a positive classification rather than the absence of an error, and a failure mode covering the general case. Verified against the shipped classifier and its evaluation harness; the rest of the page keeps its 2026-07-26 verification date.