Pattern · last reviewed 2026-07-13

Eval harness as release gate

Context & problem

You own an LLM feature in production — a support agent, an extraction pipeline, a coach. Someone opens a two-line pull request: tighten a system prompt, swap Sonnet for a cheaper model, change a retrieval chunk size. In a deterministic codebase, the test suite tells you in ninety seconds whether that diff broke anything. Here, the same diff can improve one class of inputs and quietly wreck another, and the only feedback you have is that the three outputs you eyeballed looked reasonable. So the change ships, and the regression surfaces a week later as a support ticket about the agent confidently citing a policy that does not exist.

The reason this keeps happening is structural, not careless. Correctness for a model is distributional, not binary — the same prompt produces a spread of outputs, and "did it get worse" is a claim about a distribution you cannot see by reading a handful of samples. Teams answer with "vibe-based evals" (OpenAI's own term for the failure mode): a human skims recent outputs and pronounces judgment. That does not scale past one reviewer, it is not comparable release to release, and it offers no defence against the change that helps the median case while poisoning the tail. The missing piece is not more evals sitting in a folder. It is a gate — evals wired into the release path so that "did this regress" is answered by a number the pipeline enforces, before the change reaches a user.

Forces

• Cost vs. coverage. A live LLM-judge run over hundreds of cases costs real tokens and minutes; running it on every commit is untenable. Running it never is how regressions ship. The tiers exist to resolve this tension.
• Speed vs. fidelity. Deterministic graders are instant and free but only measure what code can check. Judge-graded quality is faithful to what users feel but slow, costly, and itself fallible.
• Trust vs. the judge's own error. The gate is only as sound as the grader behind it — an uncalibrated LLM judge is a random number generator with good grammar, and a grading bug reads exactly like a model regression.
• Sensitivity vs. noise. Set the threshold too tight and normal run-to-run variance blocks good deploys; too loose and real regressions slip through. The gate must distinguish signal from the model's own non-determinism.
• Stability vs. drift. A baseline you silently re-label every time it fails stops being a baseline. The regression reference has to be frozen and defended, or the gate erodes into theatre.

The pattern

Split the harness into two tiers and place each at a different point in the release path. The offline tier is deterministic and cheap: exact-match assertions, schema validation, cosine similarity, ROUGE — everything a grader can decide without calling a model. It runs on every commit, keyless, as a required check, exactly like unit tests. Its job is to hold a regression baseline at (near) 100% and fail the build the instant something that used to work stops working.

The live tier is model-graded and expensive: an LLM judge scoring outputs against a per-dimension rubric, with a numeric acceptance threshold. It does not run on every commit — it runs when the thing it measures actually changes: a prompt edit, a model swap, a retrieval change. Its job is to gate the deploy of those changes, answering "is quality still above the line" with a number, not a glance. Between the two sits a decision that defines the whole pattern: graded rubrics where behaviour is subjective, assertions where it is checkable — and you always prefer the assertion, because it is cheaper, faster, and cannot hallucinate its own verdict.

Three supporting elements make the gate trustworthy rather than decorative. A frozen regression baseline — the set of cases that must keep passing, versioned like code, never re-labelled to make a red run green. A calibrated judge — validated against human agreement before it is allowed to block anything, with its known biases engineered around. And a feedback loop — production failures flow back into the eval set, so the gate that missed a regression today catches its recurrence tomorrow.

Reference implementation notes

The pattern is provider-neutral — the tiers and the gate are the architecture. Where the three stacks differ is in what they hand you for the graded tier.

Anthropic

Anthropic's test-and-evaluate guidance is built around this sequencing: define measurable success criteria first, then develop test cases against them, then gate on the result. Their three grader types map directly onto the tiers — code-graded and exact-match belong in the offline gate; the model-graded judge belongs in the live gate. Two of their rules are load-bearing for the gate's trustworthiness: use a different model to judge than the one that generated (self-preference bias is measured at 10–25%), and treat "more cases with automated grading" as better than "fewer cases hand-graded." The Console's built-in Evaluation tool is where you iterate rubrics and test cases before wiring them into CI; the claude-cookbooks building_evals.ipynb notebook is the worked reference for all three grader types against one dataset.

AWS

On Bedrock, the graded tier is a managed service rather than code you own. Amazon Bedrock Model Evaluation offers automatic metrics, an LLM-as-a-judge mode (generally available since March 2025) with built-in quality metrics — correctness, completeness, professional style and tone — plus responsible-AI metrics like harmfulness and answer refusal. For grounded systems there is a separate RAG evaluation for Knowledge Bases, with a retrieval-only mode (is the retrieved context relevant and complete) and a retrieval-plus-generation mode. The trade-off is the usual managed-service one: less rubric control than a judge you write, but no harness to maintain. Wire the job into your deploy pipeline as the gate; the pattern is unchanged.

Cloudflare

Cloudflare's contribution is the online half — the data source, not the pre-deploy gate. AI Gateway logs every request (prompts, responses, tokens, cost), accepts per-log feedback and scores via its patchLog API, and exposes an Evaluations surface over datasets, human feedback (open beta), and metrics; Workers Logpush streams the raw logs out for offline analysis. That production stream is exactly what the feedback loop in the diagram consumes: real failures, flagged in the gateway logs, become tomorrow's eval cases. It complements the pre-deploy gate rather than replacing it — the gate catches known regressions before release; the gateway logs surface the unknown ones after.

Trade-offs

• The harness is production code you now maintain. Graders drift, rubrics rot, judge prompts need re-tuning as models change. A grading bug is a regression too: on CORE-Bench, fixing the grader moved a reported score from 42% to 95% with no change to the model. You are signing up to debug the harness before you blame the model.
• Live gates cost tokens and wall-clock. A judge run over a real case set is minutes and dollars per deploy. That is the price of the tiering being worth it — but it means the gate has a budget, and a runaway eval set can make deploys slow enough that people route around them.
• Calibration is ongoing, not one-off. Position, verbosity, and self-preference biases are measured phenomena; keeping a judge calibrated against human spot-checks is recurring work, and a judge that drifts silently gates on the wrong thing.
• A gate invites gaming. Tune against the same cases you score against and the harness becomes a mirror — you optimise the number, not the behaviour. The held-out split and the "never paste eval examples into the prompt" rule are not optional hygiene; they are what keeps the gate honest.

When not to use this

Skip the pattern when the output is deterministically checkable end to end. If a JSON-schema validator, a compiler, or an exact-match assertion already decides pass/fail, that is your gate — ordinary tests cover it, and adding a judge on top is cost with no coverage. The pattern earns its complexity specifically where correctness is a distribution a human would otherwise eyeball.

Skip the live tier — keep only the cheap offline assertions — for a genuinely throwaway prototype whose every output a human reviews individually anyway. But be honest about "throwaway": the moment a prompt change can silently degrade something a user depends on, the exemption is over. And do not reach for this to replace production monitoring. The gate answers "did this change regress against known cases" before release; it says nothing about the novel failure a real user finds next Tuesday. That is the feedback loop's job, not the gate's — the two are complements, and treating a green gate as proof of production health is how the confident-but-wrong regression ships anyway.

As-built evidence

This platform runs the pattern on its own coach agent, on the exact stack described across this site — Cloudflare Workers + D1 + AI Gateway → OpenRouter → Claude. The offline tier is a deterministic mock-graded suite (npm run eval) that runs keyless on every change, spending nothing; it is part of the per-slice definition of done alongside typecheck and the unit tests. The live tier is a pedagogy-adherence eval that gates any change to the coach's prompt or model: it must clear an 80% threshold, median of three runs, with early exit — two runs at or above 80% pass without paying for the third. The live gate spends real tokens and therefore runs only when the prompt or model actually changes, never on an unrelated commit; the offline gate runs always. The judge-calibration reference is treated as frozen — certified against live behaviour and never re-labelled to turn a red run green, which is the discipline that keeps the 80% line meaning something. The offline/live split, the threshold, and the median-of-three rule are exactly the tiering this page prescribes, running in production rather than theorised. The full step-by-step build of that harness is documented in the eval-first development workflow.

Changelog

• 2026-07-13 — Initial publication.