Workflow · last verified 2026-07-13

Eval-first development: build the harness before the feature

The job this workflow does

An LLM feature without an eval harness cannot be changed safely. Every prompt tweak, model swap, and retrieval change becomes a leap of faith judged by whoever eyeballed the last three outputs. OpenAI's evaluation guidance names the failure mode directly — "vibe-based evals" — and Anthropic's test-and-evaluate docs open with the fix: define success criteria before you build, then develop test cases against them.

The trap is sequencing. Teams build the feature first, then bolt on evals when the first regression ships — at which point the eval set gets reverse-engineered from what the feature already does, measuring conformance to the current behavior instead of fitness for the job. Writing the harness first forces you to state what "good" means while you still can, and gives every subsequent change a number instead of a feeling.

This workflow prescribes one good way, in eight steps. We run it ourselves: on this platform, every coach prompt or deploy change is gated by an offline eval suite (npm run eval) plus a live 80% pedagogy-adherence gate. That harness existed before most of the coach's features did.

The workflow

1. Define success before writing the feature

Write the success criteria first, as Anthropic's define-success guidance prescribes: specific, measurable, achievable, relevant, and time-bound — and multidimensional. A single accuracy number hides the dimensions users actually feel: task fidelity, consistency across similar inputs, tone, latency, and price all belong on the sheet.

The test of a criterion is whether a grader could score it. "Safe outputs" is a wish; "fewer than 0.1% of 10,000 outputs flagged for toxicity" (the docs' own example) is a criterion. Rewrite every vague line until it names a threshold, a sample size, or a comparison.

# success-criteria.md — one block per dimension, each mechanically checkable
task fidelity:  ≥90% of extraction cases exact-match the reference JSON
consistency:    paraphrased duplicates agree on the answer ≥95% of pairs
tone:           LLM judge rubric "professional, no hedging" ≥4/5 median
latency:        p95 end-to-end < 3s on the eval set
price:          ≤$0.02 per request at expected volume

This document is the contract for everything that follows — the eval set instantiates it, the graders measure it, the CI gate enforces it.

2. Build the eval set from the job, not from your imagination

Start with 20–50 tasks drawn from the real job — actual user inputs, support tickets, the transcripts that made you want the feature. Anthropic's agent-evals guidance starts there for exactly this reason: invented cases test the feature you imagined; real cases test the one users get.

Include the inputs that break things, per the develop-tests docs: ambiguous phrasings, irrelevant questions, overlong inputs, and adversarial attempts. And balance positive and negative cases — a one-sided set produces one-sided optimization (an eval set full of "should trigger search" cases trains you toward an agent that always triggers search).

On sizing, the docs' operating principle is volume over polish: "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Their implied scale varies by grader: on the order of 1,000 cases for exact-match tasks, 500 for binary safety checks, 200 for ROUGE-scored summarization, 100 for Likert-graded quality. You won't have that on day one — start with the 20–50 real tasks and grow (step 8). Version the dataset like code; hold out a test split you never tune against.

3. Pick the cheapest grader that measures the criterion

Anthropic's docs describe three grader types; use the cheapest one that actually measures your criterion, in this order:

• Code-graded — exact string match, cosine similarity over sentence embeddings, ROUGE-L for summaries. Fast, objective, free to run at volume. If the criterion can be checked in code, check it in code.
• Model-graded — an LLM judge scoring a Likert 1–5 against a rubric, binary pass/fail, or an ordinal scale. Flexible enough for tone, helpfulness, adherence. The docs' explicit caution: it is "generally best practice to use a different model to evaluate than the model used to generate."
• Human — the gold standard and the most expensive. In this workflow humans are the calibration reference for the judge (step 4), not the per-run grader.

// grader ladder — reach for the next rung only when the current one can't measure it
const graders = {
  extraction:  (out, ref) => out.trim() === ref.trim(),          // exact match
  retrieval:   (out, ref) => cosine(embed(out), embed(ref)) > .9, // SBERT-style
  summary:     (out, ref) => rougeL(out, ref) > .7,
  tone:        (out) => llmJudge(RUBRIC.tone, out),               // judge model ≠ generator
};

For a worked end-to-end example of all three, Anthropic's cookbook notebook building_evals.ipynb builds each grader type against the same dataset.

4. Calibrate the judge before trusting it

An uncalibrated LLM judge is a random number generator with good grammar. Anthropic's agent-evals engineering guidance prescribes the fix: a structured rubric per dimension, an escape hatch (let the judge answer "Unknown" instead of forcing a score), and frequent calibration against expert human judgment as a closed loop — plus actually reading transcripts to verify the grader is grading what you think it is.

The benchmark for "calibrated" comes from Zheng et al. (arXiv 2306.05685): a strong judge reaching over 80% agreement with human experts matches the ~81% agreement humans reach with each other — human-level agreement is the ceiling, and a good judge can hit it. Use the most capable model available as the judge (OpenAI's best-practices guidance says the same); judge quality is not the place to save money.

Three measured biases to engineer around, from the same paper and subsequent measurements:

• Position bias — in pairwise comparisons, judges prefer the first-listed answer up to 75% of the time. Swap or randomize the order and require agreement.
• Verbosity bias — padding an answer can shift scores 0.5–1.5 points on a 5-point scale. Tight rubrics that score the criterion, not the word count.
• Self-preference — models favor their own outputs by 10–25%. Use a different judge model than the generator (the same rule as step 3, now with a number attached).

Mitigations, in the order they pay off: randomized ordering, tight per-dimension rubrics, ensemble judges where the stakes justify the cost, and ongoing calibration monitoring — log judge scores against periodic human spot-checks and alarm on drift.

5. Wire it into CI — the harness gates deploys

An eval suite that runs when someone remembers is a dashboard, not a gate. Split the suite the way Anthropic's agent-evals guidance does: capability evals start at a low pass rate and measure headroom on what you're trying to make work; regression evals hold a ~100% baseline and run continuously to catch what used to work breaking. When a capability eval saturates, it graduates into regression duty.

The offline/online split (as LangSmith's evaluation concepts frame it): offline evals run curated datasets with references pre-deploy, "as unit tests" — per-PR or nightly with thresholds; online evals run an LLM judge over production traces after deploy. This step wires the offline half; production traces feed step 8.

This is exactly how this platform ships its coach: npm run eval runs the offline deterministic suite on every change, and any coach prompt or deploy change must additionally pass a live pedagogy-adherence gate at 80% — median of three runs, with early exit (two runs at or above 80% pass without the third). The offline suite is free and runs always; the live gate spends real tokens and runs only when the prompt or model actually changed.

# package.json — the gate is a script, not a habit
"eval":      "node evals/run.mjs --suite regression --threshold 1.0",
"eval:cap":  "node evals/run.mjs --suite capability",          # tracked, not gating
"eval:live": "node evals/run.mjs --live --threshold 0.8 --median-of 3"

Treat the harness itself as production code, because grading bugs are regressions too: fixing the grading harness on CORE-Bench moved a reported Opus 4.5 score from 42% to 95% — the model hadn't changed, the grader had been wrong. Read failing transcripts before concluding the model regressed.

6. Now build the feature against the harness

With the gate in place, feature development becomes a loop: change the prompt, retrieval, or model; run the offline suite; read the failures; repeat. OpenAI's guidance compresses it to "evaluate early and often" — every iteration gets a score, and the score is comparable to yesterday's.

Comparable, that is, if you respect the error bars. Anthropic's statistical guidance ("Adding Error Bars to Evals", arXiv 2411.00640) is the corrective for celebrating noise: report the standard error of the mean alongside every score; compare variants with paired differences on the same questions, not two independent runs; and run a power analysis to size N for the effect you need to detect. The sharpest warning is about clustered questions — cluster-adjusted standard errors can be more than 3× the naive calculation, meaning unadjusted evals look far more precise than they are.

# cost-aware statistics: resample per question, pair the comparison
node evals/run.mjs --suite capability --trials 5 --paired-against baseline.json
# → mean 0.74 ± 0.03 SEM (paired Δ vs baseline +0.06, p<0.05) — now you may celebrate

The same paper's cost note: multiple samples per question tame the model's own non-determinism, and resampling per question cuts variance cheaper than adding new questions. A two-point improvement inside the error bars is not an improvement; ship it only when the paired difference clears the noise.

7. Evaluate agents on outcome and trajectory

Single-turn grading breaks down for agents. Anthropic's agent-evals guidance says to grade both halves: the outcome — the final environment state, "does the flight reservation exist in the database" — and the trajectory — the transcript of tool calls, reasoning, and turn efficiency. OpenAI's trace grading takes the same combined position: graders score the trajectory and the final answer together. Outcome tells you whether the job got done; trajectory tells you whether it got done in a way you can afford and trust.

Two rules keep agent evals honest. First, avoid rigid path-checking — agents find valid alternate routes, and a grader that demands one specific tool sequence fails correct runs. Grade what must be true, not how it must have happened. Second, run each trial in an isolated clean environment — shared state between trials inflates scores, because trial two inherits trial one's half-finished work.

// agent eval: fresh environment per trial, outcome + trajectory graded together
for (const task of tasks) {
  const env = await freshEnvironment(task.fixtures);   // never shared across trials
  const run = await agent.run(task.prompt, env);
  results.push({
    outcome:    await env.check(task.assertions),      // final state, not the transcript
    trajectory: llmJudge(RUBRIC.trajectory, run.transcript), // tool use, turns, reasoning
  });
}

And one diagnostic worth memorizing, straight from the guidance: "a 0% pass rate across many trials is most often a signal of a broken task, not an incapable agent." Debug the harness before the agent.

8. Grow and rotate the suite

The eval set is a living artifact. Production failures are its best source — OpenAI's guidance says to grow the eval set from them, and hold out reference transcripts you never tune against. Every incident, bad trace flagged by the online judge, and user complaint becomes a candidate case; keep the positive/negative balance as you add.

Rotate on saturation: an eval pinned at 100% produces zero signal about improvement — graduate it to the regression suite (where 100% is the point) and replace it with a harder capability eval. And watch for contamination as public benchmarks age into training data: roughly 29% of MMLU items show contamination signals, and cleaning contaminated GSM8K items dropped one model's score by up to 13 points (arXiv 2406.04244). The application-level analogue is self-inflicted: don't paste your eval examples into the system prompt — you are training-set-contaminating your own harness.

# the recurring loop — calendar, not incident
node evals/triage.mjs --from production-traces --since 30d   # candidates from real failures
node evals/run.mjs --suite capability --report saturation    # ≥98% for 3 releases → rotate

Artifacts this workflow produces

• The success-criteria document — SMART, multidimensional, each line mechanically checkable (step 1).
• The versioned eval dataset — real tasks plus edge and adversarial cases, balanced, with a held-out test split.
• Grader implementations — code graders where possible, judge rubrics with escape hatches where not, a different model judging than generating.
• The CI gate config — regression suite at ~100% threshold gating deploys; capability suite tracked; live gate for prompt/model changes.
• The calibration log — judge scores vs. human spot-checks over time, with the bias mitigations in force.

Pitfalls

• Overfitting to the eval set. Tuning against the same cases you score against turns the harness into a mirror. Hold out a test split (develop-tests docs) and never paste eval examples into the system prompt.
• Rigid path-checking graders. Agents find valid alternate routes; a grader that demands one tool sequence fails correct runs. Grade outcomes and trajectory quality, not exact paths.
• Celebrating differences inside the error bars. Small-N score differences are routinely within noise (arXiv 2411.00640) — and cluster-adjusted standard errors can be >3× naive. Paired comparisons, SEM reported, power analysis to size N.
• Saturated evals left in place. A 100% eval measures nothing. Rotate it to regression duty and add headroom.
• Trusting an uncalibrated judge. Position, verbosity, and self-preference biases are measured phenomena, not hypotheticals. Calibrate against humans before the judge gates anything.
• Grading bugs read as model regressions. The CORE-Bench fix moved a score from 42% to 95% without touching the model. When a score drops, read the transcripts before blaming the model.
• Benchmark contamination. Public benchmark items leak into training data (~29% of MMLU shows signals); a public benchmark score is not evidence about your task. Your own eval set, from your own job, is.

When not to use this workflow

Skip the harness when the output is deterministically checkable end to end — if a JSON schema validator or a compiler already tells you pass/fail, that is your eval, and ordinary tests cover it. Skip the judge machinery (steps 4 and parts of 5) for a genuinely throwaway prototype whose outputs a human reviews individually anyway — but be honest about "throwaway": the moment a prompt change can silently degrade something a user depends on, you are past the exemption.

And don't reach for this page to learn what evals are — grader taxonomies, metric definitions, and the benchmark landscape live in the LLM evaluation guide. This page assumes that vocabulary and prescribes the practice.

Changelog

• 2026-07-13 — initial version, verified against Anthropic test-and-evaluate docs and 2026 engineering guidance.