Pattern · last reviewed 2026-08-10

Eval harness as release gate

This pattern is the architecture decision behind layer 5 of the AI quality stack: evals. Guardrails and HITL gates catch a single bad output or a risky action; the eval harness is the only layer that tells you whether a change made the system better or worse across the whole case distribution — a number, not a glance at three outputs.

Part of the AI quality stack — the layered gate chain for knowing your LLM is delivering quality.

Context & problem

You own an LLM feature in production — a support agent, an extraction pipeline, a coach. Someone opens a two-line pull request: tighten a system prompt, swap Sonnet for a cheaper model, change a retrieval chunk size. In a deterministic codebase, the test suite tells you in ninety seconds whether that diff broke anything. Here, the same diff can improve one class of inputs and quietly wreck another, and the only feedback you have is that the three outputs you eyeballed looked reasonable. So the change ships, and the regression surfaces a week later as a support ticket about the agent confidently citing a policy that does not exist.

The reason this keeps happening is structural, not careless. Correctness for a model is distributional, not binary — the same prompt produces a spread of outputs, and "did it get worse" is a claim about a distribution you cannot see by reading a handful of samples. Teams answer with "vibe-based evals" (OpenAI's own term for the failure mode): a human skims recent outputs and pronounces judgment. That does not scale past one reviewer, it is not comparable release to release, and it offers no defence against the change that helps the median case while poisoning the tail. The missing piece is not more evals sitting in a folder. It is a gate — evals wired into the release path so that "did this regress" is answered by a number the pipeline enforces, before the change reaches a user.

Forces

• Cost vs. coverage. A live LLM-judge run over hundreds of cases costs real tokens and minutes; running it on every commit is untenable. Running it never is how regressions ship. The tiers exist to resolve this tension.
• Speed vs. fidelity. Deterministic graders are instant and free but only measure what code can check. Judge-graded quality is faithful to what users feel but slow, costly, and itself fallible.
• Trust vs. the judge's own error. The gate is only as sound as the grader behind it — an uncalibrated LLM judge is a random number generator with good grammar, and a grading bug reads exactly like a model regression.
• Sensitivity vs. noise. Set the threshold too tight and normal run-to-run variance blocks good deploys; too loose and real regressions slip through. The gate must distinguish signal from the model's own non-determinism.
• Stability vs. drift. A baseline you silently re-label every time it fails stops being a baseline. The regression reference has to be frozen and defended, or the gate erodes into theatre.

The pattern

Split the harness into two tiers and place each at a different point in the release path. The offline tier is deterministic and cheap: exact-match assertions, schema validation, cosine similarity, ROUGE — everything a grader can decide without calling a model. It runs on every commit, keyless, as a required check, exactly like unit tests. Its job is to hold a regression baseline at (near) 100% and fail the build the instant something that used to work stops working.

The live tier is model-graded and expensive: an LLM judge scoring outputs against a per-dimension rubric, with a numeric acceptance threshold. It does not run on every commit — it runs when the thing it measures actually changes: a prompt edit, a model swap, a retrieval change. Its job is to gate the deploy of those changes, answering "is quality still above the line" with a number, not a glance. Between the two sits a decision that defines the whole pattern: graded rubrics where behaviour is subjective, assertions where it is checkable — and you always prefer the assertion, because it is cheaper, faster, and cannot hallucinate its own verdict.

Three supporting elements make the gate trustworthy rather than decorative. A frozen regression baseline — the set of cases that must keep passing, versioned like code, never re-labelled to make a red run green. A calibrated judge — validated against human agreement before it is allowed to block anything, with its known biases engineered around. And a feedback loop — production failures flow back into the eval set, so the gate that missed a regression today catches its recurrence tomorrow.

Reference implementation notes

The pattern is provider-neutral — the tiers and the gate are the architecture. Where the three stacks differ is in what they hand you for the graded tier.

Anthropic

Anthropic's test-and-evaluate guidance is built around this sequencing: define measurable success criteria first, then develop test cases against them, then gate on the result. Their three grader types map directly onto the tiers — code-graded and exact-match belong in the offline gate; the model-graded judge belongs in the live gate. Two of their rules are load-bearing for the gate's trustworthiness: use a different model to judge than the one that generated (self-preference bias is measured at 10–25%), and treat "more cases with automated grading" as better than "fewer cases hand-graded." The Console's built-in Evaluation tool is where you iterate rubrics and test cases before wiring them into CI; the claude-cookbooks building_evals.ipynb notebook is the worked reference for all three grader types against one dataset.

AWS

On Bedrock, the graded tier is a managed service rather than code you own. Amazon Bedrock Model Evaluation offers automatic metrics, an LLM-as-a-judge mode (generally available since March 2025) with built-in quality metrics — correctness, completeness, professional style and tone — plus responsible-AI metrics like harmfulness and answer refusal. For grounded systems there is a separate RAG evaluation for Knowledge Bases, with a retrieval-only mode (is the retrieved context relevant and complete) and a retrieval-plus-generation mode. The trade-off is the usual managed-service one: less rubric control than a judge you write, but no harness to maintain. Wire the job into your deploy pipeline as the gate; the pattern is unchanged.

Cloudflare

Cloudflare's contribution is the online half — the data source, not the pre-deploy gate. AI Gateway logs every request (prompts, responses, tokens, cost), accepts per-log feedback and scores via its patchLog API, and exposes an Evaluations surface over datasets, human feedback (open beta), and metrics; Workers Logpush streams the raw logs out for offline analysis. That production stream is exactly what the feedback loop in the diagram consumes: real failures, flagged in the gateway logs, become tomorrow's eval cases. It complements the pre-deploy gate rather than replacing it — the gate catches known regressions before release; the gateway logs surface the unknown ones after.

Trade-offs

• The harness is production code you now maintain. Graders drift, rubrics rot, judge prompts need re-tuning as models change. A grading bug is a regression too: on CORE-Bench, fixing the grader moved a reported score from 42% to 95% with no change to the model. You are signing up to debug the harness before you blame the model.
• Live gates cost tokens and wall-clock. A judge run over a real case set is minutes and dollars per deploy. That is the price of the tiering being worth it — but it means the gate has a budget, and a runaway eval set can make deploys slow enough that people route around them.
• Calibration is ongoing, not one-off. Position, verbosity, and self-preference biases are measured phenomena; keeping a judge calibrated against human spot-checks is recurring work, and a judge that drifts silently gates on the wrong thing.
• A gate invites gaming. Tune against the same cases you score against and the harness becomes a mirror — you optimise the number, not the behaviour. The held-out split and the "never paste eval examples into the prompt" rule are not optional hygiene; they are what keeps the gate honest.

When not to use this

Skip the pattern when the output is deterministically checkable end to end. If a JSON-schema validator, a compiler, or an exact-match assertion already decides pass/fail, that is your gate — ordinary tests cover it, and adding a judge on top is cost with no coverage. The pattern earns its complexity specifically where correctness is a distribution a human would otherwise eyeball.

Skip the live tier — keep only the cheap offline assertions — for a genuinely throwaway prototype whose every output a human reviews individually anyway. But be honest about "throwaway": the moment a prompt change can silently degrade something a user depends on, the exemption is over. And do not reach for this to replace production monitoring. The gate answers "did this change regress against known cases" before release; it says nothing about the novel failure a real user finds next Tuesday. That is the feedback loop's job, not the gate's — the two are complements, and treating a green gate as proof of production health is how the confident-but-wrong regression ships anyway.

As-built evidence

This platform runs the pattern on its own coach agent, on the exact stack described across this site — Cloudflare Workers + D1 + AI Gateway → OpenRouter → Claude. The offline tier is a deterministic mock-graded suite (npm run eval) that runs keyless on every change, spending nothing; it is part of the per-slice definition of done alongside typecheck and the unit tests. The live tier is a pedagogy-adherence eval that gates any change to the coach's prompt or model: it must clear an 80% threshold, median of three runs, with early exit — two runs at or above 80% pass without paying for the third. The live gate spends real tokens and therefore runs only when the prompt or model actually changes, never on an unrelated commit; the offline gate runs always. The judge-calibration reference is treated as frozen — certified against live behaviour and never re-labelled to turn a red run green, which is the discipline that keeps the 80% line meaning something. The offline/live split, the threshold, and the median-of-three rule are exactly the tiering this page prescribes, running in production rather than theorised. The full step-by-step build of that harness is documented in the eval-first development workflow.

The median-of-three rule is a measurement, not a preference — and the measurement is embarrassing. The gate was single-run first. During one coach release, three consecutive live runs against identical code scored 74.1% (fail), then a pass, then 81.5% (pass): the same commit landing on both sides of the 80% line three times in a row. A gate that flips on unchanged input is not measuring the change, it is measuring the sampled system's own variance — here the coach model's, because this gate has no judge in it: every check in src/evals/coach.eval.ts is a deterministic predicate over the recorded transcript — text, tool names, events, stop reason — and the judge hook is defaulted off. (The grader gate below does use a judge, and there the same rule would be absorbing both sources at once.) At the boundary the flapping goes in both directions, blocking good releases on a flaky low run and shipping a bad prompt on a lucky high one. What replaced it is up to three runs per invocation: pass at two runs at or above 80%, fail at two below, early exit the moment the verdict is decided, so the typical cost is two runs rather than three. Best-of-three was considered and rejected on the arithmetic recorded alongside the decision — a genuinely sub-bar prompt would clear it roughly two times in three, which loosens the gate instead of stabilising it. The 80% bar and every individual check stayed frozen; only the sampling changed, because the sampling was the part that was wrong. If your threshold gate has never been run twice on the same commit, you do not yet know whether you have this problem.

The gate has caught the judge, not just the code. A second live gate covers the short-answer grader: per-key-point agreement with human labels at or above 80%, plus a no-reward condition — no fixture written specifically to game the rubric may be graded correct (adversarialPassedAsCorrect must come back empty in src/evals/grader.eval.ts). Its first live run failed that second condition. The fixture was a keyword salad: every phrase from that item's rubric reproduced verbatim, strung into grammatical sentences that assert nothing about any of them, ending on a line about how important these concepts are. The judge credited it. The fix went into the judge's prompt rather than the model or the fixture: rule 1 of buildShortAnswerPrompt (src/lib/grade.ts) now states that naming a concept without asserting anything about it is not a hit, and that the supporting quote has to be a statement rather than a word list. The re-run passed. Two things are worth taking from it. Adversarial cases belong inside the gate, because a rubric-shaped attack scores well on every honest aggregate you would otherwise watch — mean score would have gone up. And the fix landing in the grader prompt is the first trade-off above arriving in practice: the harness is production code, and a gate reading wrong makes the harness the first suspect, not the model.

One honest limit on all of the above: both live gates are invoked by hand before a release, not by CI, because each invocation is real token spend on a solo budget. That is a discipline, not an enforcement — the thing that physically cannot be skipped here is the offline tier, which is why it carries the always-on load and why the regression baseline lives there rather than behind the judge.

Changelog

• 2026-07-13 — Initial publication.
• 2026-08-10 — Added the numbers behind the median-of-three rule (74.1% / pass / 81.5% on identical code, and why best-of-three was rejected), the adversarial no-reward gate that failed its first live run when the judge credited a keyword salad, and an honest note that both live gates are hand-invoked rather than enforced by CI.
• 2026-08-15 — Named LLM evaluation in the opening paragraph; the pattern name, id and URL are unchanged.