Workflow · last verified 2026-07-13

Eval-first development: build the harness before the feature

A response can be well-formed, on-brief, and still wrong on cases you didn't happen to eyeball — spec-first framing and structured output only catch the malformed or off-brief kind. Closing that gap is what a scored harness is for: layer 5 of the AI quality stack, evals.

Part of the AI quality stack — the layered gate chain for knowing your LLM is delivering quality.

The job this workflow does

An LLM feature without an eval harness cannot be changed safely. Every prompt tweak, model swap, and retrieval change becomes a leap of faith judged by whoever eyeballed the last three outputs. OpenAI's evaluation guidance names the failure mode directly — "vibe-based evals" — and Anthropic's test-and-evaluate docs open with the fix: define success criteria before you build, then develop test cases against them.

The trap is sequencing. Teams build the feature first, then bolt on evals when the first regression ships — at which point the eval set gets reverse-engineered from what the feature already does, measuring conformance to the current behavior instead of fitness for the job. Writing the harness first forces you to state what "good" means while you still can, and gives every subsequent change a number instead of a feeling.

This workflow prescribes one good way, in eight steps. We run it ourselves: on this platform, every coach prompt or deploy change is gated by an offline eval suite (npm run eval) plus a live 80% pedagogy-adherence gate. That harness existed before most of the coach's features did.

The workflow

1. Define success before writing the feature

Write the success criteria first, as Anthropic's define-success guidance prescribes: specific, measurable, achievable, relevant, and time-bound — and multidimensional. A single accuracy number hides the dimensions users actually feel: task fidelity, consistency across similar inputs, tone, latency, and price all belong on the sheet.

The test of a criterion is whether a grader could score it. "Safe outputs" is a wish; "fewer than 0.1% of 10,000 outputs flagged for toxicity" (the docs' own example) is a criterion. Rewrite every vague line until it names a threshold, a sample size, or a comparison.

# success-criteria.md — one block per dimension, each mechanically checkable
task fidelity:  ≥90% of extraction cases exact-match the reference JSON
consistency:    paraphrased duplicates agree on the answer ≥95% of pairs
tone:           LLM judge rubric "professional, no hedging" ≥4/5 median
latency:        p95 end-to-end < 3s on the eval set
price:          ≤$0.02 per request at expected volume

This document is the contract for everything that follows — the eval set instantiates it, the graders measure it, the CI gate enforces it.

2. Build the eval set from the job, not from your imagination

Start with 20–50 tasks drawn from the real job — actual user inputs, support tickets, the transcripts that made you want the feature. Anthropic's agent-evals guidance starts there for exactly this reason: invented cases test the feature you imagined; real cases test the one users get.

Include the inputs that break things, per the develop-tests docs: ambiguous phrasings, irrelevant questions, overlong inputs, and adversarial attempts. And balance positive and negative cases — a one-sided set produces one-sided optimization (an eval set full of "should trigger search" cases trains you toward an agent that always triggers search).

On sizing, the docs' operating principle is volume over polish: "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Their implied scale varies by grader: on the order of 1,000 cases for exact-match tasks, 500 for binary safety checks, 200 for ROUGE-scored summarization, 100 for Likert-graded quality. You won't have that on day one — start with the 20–50 real tasks and grow (step 8). Version the dataset like code; hold out a test split you never tune against.

3. Pick the cheapest grader that measures the criterion

Anthropic's docs describe three grader types; use the cheapest one that actually measures your criterion, in this order:

• Code-graded — exact string match, cosine similarity over sentence embeddings, ROUGE-L for summaries. Fast, objective, free to run at volume. If the criterion can be checked in code, check it in code.
• Model-graded — an LLM judge scoring a Likert 1–5 against a rubric, binary pass/fail, or an ordinal scale. Flexible enough for tone, helpfulness, adherence. The docs' explicit caution: it is "generally best practice to use a different model to evaluate than the model used to generate."
• Human — the gold standard and the most expensive. In this workflow humans are the calibration reference for the judge (step 4), not the per-run grader.

// grader ladder — reach for the next rung only when the current one can't measure it
const graders = {
  extraction:  (out, ref) => out.trim() === ref.trim(),          // exact match
  retrieval:   (out, ref) => cosine(embed(out), embed(ref)) > .9, // SBERT-style
  summary:     (out, ref) => rougeL(out, ref) > .7,
  tone:        (out) => llmJudge(RUBRIC.tone, out),               // judge model ≠ generator
};

For a worked end-to-end example of all three, Anthropic's cookbook notebook building_evals.ipynb builds each grader type against the same dataset.

4. Calibrate the judge before trusting it

An uncalibrated LLM judge is a random number generator with good grammar. Anthropic's agent-evals engineering guidance prescribes the fix: a structured rubric per dimension, an escape hatch (let the judge answer "Unknown" instead of forcing a score), and frequent calibration against expert human judgment as a closed loop — plus actually reading transcripts to verify the grader is grading what you think it is.

The benchmark for "calibrated" comes from Zheng et al. (arXiv 2306.05685): a strong judge reaching over 80% agreement with human experts matches the ~81% agreement humans reach with each other — human-level agreement is the ceiling, and a good judge can hit it. Use the most capable model available as the judge (OpenAI's best-practices guidance says the same); judge quality is not the place to save money.

Three measured biases to engineer around, from the same paper and subsequent measurements:

• Position bias — in pairwise comparisons, judges prefer the first-listed answer up to 75% of the time. Swap or randomize the order and require agreement.
• Verbosity bias — padding an answer can shift scores 0.5–1.5 points on a 5-point scale. Tight rubrics that score the criterion, not the word count.
• Self-preference — models favor their own outputs by 10–25%. Use a different judge model than the generator (the same rule as step 3, now with a number attached).

Mitigations, in the order they pay off: randomized ordering, tight per-dimension rubrics, ensemble judges where the stakes justify the cost, and ongoing calibration monitoring — log judge scores against periodic human spot-checks and alarm on drift.

5. Wire it into CI — the harness gates deploys

An eval suite that runs when someone remembers is a dashboard, not a gate. Split the suite the way Anthropic's agent-evals guidance does: capability evals start at a low pass rate and measure headroom on what you're trying to make work; regression evals hold a ~100% baseline and run continuously to catch what used to work breaking. When a capability eval saturates, it graduates into regression duty.

The offline/online split (as LangSmith's evaluation concepts frame it): offline evals run curated datasets with references pre-deploy, "as unit tests" — per-PR or nightly with thresholds; online evals run an LLM judge over production traces after deploy. This step wires the offline half; production traces feed step 8.

This is exactly how this platform ships its coach: npm run eval runs the offline deterministic suite on every change, and any coach prompt or deploy change must additionally pass a live pedagogy-adherence gate at 80% — median of three runs, with early exit (two runs at or above 80% pass without the third). The offline suite is free and runs always; the live gate spends real tokens and runs only when the prompt or model actually changed.

# package.json — the gate is a script, not a habit
"eval":      "node evals/run.mjs --suite regression --threshold 1.0",
"eval:cap":  "node evals/run.mjs --suite capability",          # tracked, not gating
"eval:live": "node evals/run.mjs --live --threshold 0.8 --median-of 3"

Treat the harness itself as production code, because grading bugs are regressions too: fixing the grading harness on CORE-Bench moved a reported Opus 4.5 score from 42% to 95% — the model hadn't changed, the grader had been wrong. Read failing transcripts before concluding the model regressed.

6. Now build the feature against the harness

With the gate in place, feature development becomes a loop: change the prompt, retrieval, or model; run the offline suite; read the failures; repeat. OpenAI's guidance compresses it to "evaluate early and often" — every iteration gets a score, and the score is comparable to yesterday's.

Comparable, that is, if you respect the error bars. Anthropic's statistical guidance ("Adding Error Bars to Evals", arXiv 2411.00640) is the corrective for celebrating noise: report the standard error of the mean alongside every score; compare variants with paired differences on the same questions, not two independent runs; and run a power analysis to size N for the effect you need to detect. The sharpest warning is about clustered questions — cluster-adjusted standard errors can be more than 3× the naive calculation, meaning unadjusted evals look far more precise than they are.

# cost-aware statistics: resample per question, pair the comparison
node evals/run.mjs --suite capability --trials 5 --paired-against baseline.json
# → mean 0.74 ± 0.03 SEM (paired Δ vs baseline +0.06, p<0.05) — now you may celebrate

The same paper's cost note: multiple samples per question tame the model's own non-determinism, and resampling per question cuts variance cheaper than adding new questions. A two-point improvement inside the error bars is not an improvement; ship it only when the paired difference clears the noise.

7. Evaluate agents on outcome and trajectory

Single-turn grading breaks down for agents. Anthropic's agent-evals guidance says to grade both halves: the outcome — the final environment state, "does the flight reservation exist in the database" — and the trajectory — the transcript of tool calls, reasoning, and turn efficiency. OpenAI's trace grading takes the same combined position: graders score the trajectory and the final answer together. Outcome tells you whether the job got done; trajectory tells you whether it got done in a way you can afford and trust.

Two rules keep agent evals honest. First, avoid rigid path-checking — agents find valid alternate routes, and a grader that demands one specific tool sequence fails correct runs. Grade what must be true, not how it must have happened. Second, run each trial in an isolated clean environment — shared state between trials inflates scores, because trial two inherits trial one's half-finished work.

// agent eval: fresh environment per trial, outcome + trajectory graded together
for (const task of tasks) {
  const env = await freshEnvironment(task.fixtures);   // never shared across trials
  const run = await agent.run(task.prompt, env);
  results.push({
    outcome:    await env.check(task.assertions),      // final state, not the transcript
    trajectory: llmJudge(RUBRIC.trajectory, run.transcript), // tool use, turns, reasoning
  });
}

And one diagnostic worth memorizing, straight from the guidance: "a 0% pass rate across many trials is most often a signal of a broken task, not an incapable agent." Debug the harness before the agent.

8. Grow and rotate the suite

The eval set is a living artifact. Production failures are its best source — OpenAI's guidance says to grow the eval set from them, and hold out reference transcripts you never tune against. Every incident, bad trace flagged by the online judge, and user complaint becomes a candidate case; keep the positive/negative balance as you add.

Rotate on saturation: an eval pinned at 100% produces zero signal about improvement — graduate it to the regression suite (where 100% is the point) and replace it with a harder capability eval. And watch for contamination as public benchmarks age into training data: roughly 29% of MMLU items show contamination signals, and cleaning contaminated GSM8K items dropped one model's score by up to 13 points (arXiv 2406.04244). The application-level analogue is self-inflicted: don't paste your eval examples into the system prompt — you are training-set-contaminating your own harness.

# the recurring loop — calendar, not incident
node evals/triage.mjs --from production-traces --since 30d   # candidates from real failures
node evals/run.mjs --suite capability --report saturation    # ≥98% for 3 releases → rotate

As built: someone else's effect is not your result

Step 1 is easy to satisfy when the feature is your own idea, because you have to invent the definition of good from scratch and the blank page forces the work. It is much easier to skip when the feature is justified by published research, because the paper appears to have already defined good and attached a number to it. That is the case this section is about.

On 2026-07-07 this platform shipped mastery-adaptive routing on the module page. The justification was the expertise-reversal effect — instructional scaffolding that measurably helps learners new to a topic measurably hurts learners who already partly know it, the two groups getting opposite-signed effects. Well replicated, and directly relevant to a product that had been tracking per-objective mastery for months while treating every learner identically. What shipped is small: on a fresh entry, a learner whose mean per-objective mastery for the module is at or above 0.5 lands on retrieval practice first, everyone else keeps the lesson-first path, and nothing is locked either way — the other path stays one click away.

The spec did the eval-first things right, on paper. Above the feature description sits an "Evidence honesty" note saying the routing rule is a faithful application of a strong interaction effect but that our implementation of it is unvalidated. It names a guard-rail metric rather than a win condition: the problem-first cohort's unassisted accuracy on later module items must not be worse. And the instrumentation shipped in the same change rather than a later one — two allowlisted funnel events, route_problem_first and route_lesson_first, logging best-effort which default path a fresh entry took.

It still does not work, and the reason is worth more than the intention. Those events land in an append-only table that is deliberately PII-free: (id, name, path, ref, ts), where path is location.pathname with the query string dropped on purpose. There is no learner key — and because the module page is /module.html?id=<id>, not even the module id survives. So what shipped is two anonymous counters of routed page-entries. The per-learner cohort join the guard-rail asks for cannot be computed from them at all.

# the criterion, as written in the spec
guard-rail:  problem-first cohort's unassisted accuracy on later module
             items is NOT WORSE than the lesson-first cohort
instrument:  route_problem_first / route_lesson_first, on fresh entry

# the row those events actually write
funnel_events(id, name, path, ref, ts)   -- path = pathname; no query string

# no learner key, no lesson id -> two anonymous counters.
# the cohort join the criterion needs is not computable from this.

Nor is the fix a column. The table is PII-free by an explicit logged decision about where attribution is allowed to live, so closing the gap means choosing a different instrument or narrowing the criterion to something the anonymous counters can actually answer — a decision to take deliberately, not a patch to slip in. Meanwhile the honest status of the feature is: shipped, plausible, unmeasured.

So the transferable rule is narrower and harder than "write the criterion first". A criterion is not wired until you have written the query that computes it and run it against the rows your instrumentation is actually writing. Step 1's test — could a grader score this? — has an operational twin that catches what it misses: could you run this metric today, on the data you are already collecting, and get a number? If the answer needs a schema change, a new key, or a decision, you have a stated intention rather than a gate, and the gap will surface at the worst moment — when someone finally asks whether the feature worked.

Two smaller notes from the same change. A guard-rail phrased "must not be worse" rather than "improves by X" is deliberate: a published effect is a hypothesis about your users, measured on a population that is not yours through an intervention that is not the one you built, so the honest first question is whether your version does harm — and the phrasing removes the incentive to go hunting for a favourable read. And the coach-side half of this change sits behind a live pedagogy-adherence gate that spends real money and runs by hand; the spec does not count the prompt change as validated until it passes, which is the one part of this story that is a gate rather than a wish.

Artifacts this workflow produces

• The success-criteria document — SMART, multidimensional, each line mechanically checkable (step 1).
• The versioned eval dataset — real tasks plus edge and adversarial cases, balanced, with a held-out test split.
• Grader implementations — code graders where possible, judge rubrics with escape hatches where not, a different model judging than generating.
• The CI gate config — regression suite at ~100% threshold gating deploys; capability suite tracked; live gate for prompt/model changes.
• The calibration log — judge scores vs. human spot-checks over time, with the bias mitigations in force.

Pitfalls

• Overfitting to the eval set. Tuning against the same cases you score against turns the harness into a mirror. Hold out a test split (develop-tests docs) and never paste eval examples into the system prompt.
• Rigid path-checking graders. Agents find valid alternate routes; a grader that demands one tool sequence fails correct runs. Grade outcomes and trajectory quality, not exact paths.
• Celebrating differences inside the error bars. Small-N score differences are routinely within noise (arXiv 2411.00640) — and cluster-adjusted standard errors can be >3× naive. Paired comparisons, SEM reported, power analysis to size N.
• Saturated evals left in place. A 100% eval measures nothing. Rotate it to regression duty and add headroom.
• Trusting an uncalibrated judge. Position, verbosity, and self-preference biases are measured phenomena, not hypotheticals. Calibrate against humans before the judge gates anything.
• Grading bugs read as model regressions. The CORE-Bench fix moved a score from 42% to 95% without touching the model. When a score drops, read the transcripts before blaming the model.
• Treating a published effect as a shipped result. A paper's effect size measures its intervention on its population, not yours. The success criterion for a research-justified feature is a guard-rail on your own users, and it ships in the same commit as the feature or the evidence never ships at all.
• A criterion nobody has run the query for. "Mechanically checkable" is not the same as checkable against the rows you are writing. If computing the metric needs a key your event table does not carry, you have an intention, not a gate — run it once, on real data, the day you write it.
• Benchmark contamination. Public benchmark items leak into training data (~29% of MMLU shows signals); a public benchmark score is not evidence about your task. Your own eval set, from your own job, is.

When not to use this workflow

Skip the harness when the output is deterministically checkable end to end — if a JSON schema validator or a compiler already tells you pass/fail, that is your eval, and ordinary tests cover it. Skip the judge machinery (steps 4 and parts of 5) for a genuinely throwaway prototype whose outputs a human reviews individually anyway — but be honest about "throwaway": the moment a prompt change can silently degrade something a user depends on, you are past the exemption.

And don't reach for this page to learn what evals are — grader taxonomies, metric definitions, and the benchmark landscape live in the LLM evaluation guide. This page assumes that vocabulary and prescribes the practice.

Changelog

• 2026-08-10 — added the as-built section on research-justified features: the guard-rail metric shipped alongside this platform's mastery-adaptive routing, and why its cohort join is not computable from the anonymous event rows the instrumentation actually writes; two matching pitfalls and a provenance line.
• 2026-07-13 — initial version, verified against Anthropic test-and-evaluate docs and 2026 engineering guidance.