Pattern · last reviewed 2026-10-19

Parse the Verdict, Not the Text: Model Output at a Trust Boundary

Context & problem

Anywhere your code acts on a model's answer, the answer has stopped being prose and become a decision input: a classifier that gates a request, a grader that decides pass or fail, an extractor that fills a database row. It crosses a trust boundary on the way in. The producer is probabilistic, the reply can be shaped by content you do not control, and nothing promises its form. Treat it the way you would treat a request body from the internet.

Four different things go wrong, and they are different problems:

• Malformed. Not JSON, JSON wrapped in prose, JSON preceded by reasoning, JSON cut off mid-object.
• Wrong shape. Valid JSON with a missing field, a string where a number belongs, a value outside the range you defined.
• Right shape, wrong content. Every field present and typed, and the content breaks a rule only your code can check: an "evidence" quote that is not in the text it claims to quote, a rubric item the model skipped, a justification that is empty where a verdict requires one.
• No verdict at all. A refusal, a truncation, a timeout, a provider error.

The default handling is a try/catch around JSON.parse that returns an empty object or a zero. That default is the defect. An empty object has no verdict field, the field reads as undefined, undefined is falsy, and falsy reads as "no violation" or "not correct". The system has turned a non-answer into an answer without anyone deciding to.

Forces

• Schema mode guarantees shape. It does not guarantee truth, and it does not guarantee delivery. Anthropic's structured outputs constrain generation to your schema, which removes most of the malformed and wrong-shape cases. The same documentation lists where the output can still miss the schema: a refusal comes back as a normal 200 response whose output may not match, a reply cut at the token limit can be incomplete, and enum values can differ from your schema in capitalization with no error. OpenAI's guide makes the matching distinction between its JSON mode, which yields valid JSON, and Structured Outputs, which yields schema adherence, and names refusal and a hit token limit as cases where the reply may not match. Use the mode to shrink two failure classes. Do not use it as the validation step.
• Schema mode has its own request-time failures. On Anthropic's API an unsupported keyword is a 400, not a silent no-op: recursive schemas, numeric bounds, minLength/maxLength, array constraints beyond a minimum of 0 or 1, and any additionalProperties other than false. A schema that is too complex to compile is a 400 as well. Combining it with citations is a 400. These fail loudly on the first call, which is the good kind of failure. A related one bites people who reach for tool use as the portable route to JSON: as of 2026-10-19, forcing a tool with tool_choice of any or tool returns a 400 on four current models (Claude Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1), while auto and none are accepted. The documented alternatives are strict tool use under auto, or structured outputs when the response itself needs a fixed shape.
• The rules that matter most are the ones a schema cannot state. "A reason is required when the verdict is a violation" is a relationship between two fields. "This quote must appear in the learner's answer" refers to text the schema never sees. Even the humble "non-empty string" is on the unsupported list, because minLength is. These rules live in code that holds the source and the context.
• A text-mode reply carries more than the verdict. Preamble, a restatement of the input, leaked reasoning, a closing remark. The verdict is the last thing the model says. And when the reply quotes the input, some of the bytes your parser reads were chosen by whoever wrote the input.
• "No valid verdict" costs different things to different consumers. For a security gate it can mean an unscreened request. For a grader it can mean a wrong mark on someone's progress. For an extractor it can mean a row nobody should write. One policy cannot be right for all three.

The pattern

Three decisions, in order. Each one is made once, in one place, and each has a name in the code.

1. Extract. Where the vendor offers constrained output on your model and route, use it: it removes the extraction problem. Where it does not, take the last JSON object in the reply that parses, scanning from the right, not the span from the first brace to the last. The verdict comes last; whatever precedes it is reasoning, quotation or preamble, and any brace in that material breaks a first-to-last slice. Why this matters on a security gate, and how it was found in review, is told once, in the review workflow. This page keeps only the design rule.
2. Validate the contract, not just the type. Compare the verdict field by strict equality against the values you allow, so a string, a boolean or an out-of-range number is rejected rather than coerced. Check the cross-field rules. Check every model claim that points at a source against the source, deterministically: this is a deterministic gate on a probabilistic producer, with the model's output as the thing being gated. Compare against the exact text the model was shown, which may be a sanitized copy of what the user sent, or honest answers will fail the check. Where you coerce at all, coerce toward the outcome that does not act.
3. Give "no valid verdict" its own state, then choose its policy per consumer. The state must not be representable as a negative verdict: a distinct union member, a nullable result, an explicit error object. This is fail empty applied to model output: absence must not render as a value. Then pick what the state does. Four honest options: fail closed (deny); degrade to a human or self-service path and record nothing; retry once, then degrade; or fail open on purpose, behind a circuit breaker, for a control that is one layer among several. Whichever you pick, count it. An unparseable reply still cost tokens, and a rate you cannot read is the same silence with more code.

Reference implementation notes

Extract: the last object that parses

A sketch of the extractor and a validator for a two-field verdict, adapted from the coach's input gate (the as-built section below has the real files). It is a sketch, not our code: the shipped extractor also tries one parse anchored on the last {"violation": before this scan, so braces echoed inside the verdict's own reason cost nothing as long as the verdict opens with its violation key and no stray closing brace follows it, and it caps at 64 parses, not the 200 shown here. Check the sketch against your own reply format before you rely on it.

type Verdict = { violation: 0 | 1; reason: string };

function lastJsonObject(text: string, maxTries = 200): unknown {
  let tries = 0;
  let end = text.lastIndexOf('}');
  while (end !== -1) {
    let start = text.lastIndexOf('{', end);
    while (start !== -1) {
      if (++tries > maxTries) return null;   // give up: the caller treats null as no verdict
      try { return JSON.parse(text.slice(start, end + 1)); } catch { /* try an earlier brace */ }
      // lastIndexOf clamps a negative position to 0, so stepping to start - 1
      // from index 0 finds index 0 again: stop explicitly.
      start = start === 0 ? -1 : text.lastIndexOf('{', start - 1);
    }
    end = end === 0 ? -1 : text.lastIndexOf('}', end - 1);
  }
  return null;
}

function parseVerdict(text: string): Verdict | null {
  const o = lastJsonObject(text) as { violation?: unknown; reason?: unknown } | null;
  if (!o || typeof o !== 'object') return null;
  if (o.violation !== 0 && o.violation !== 1) return null;   // strict: no "1", no true
  const reason = typeof o.reason === 'string' ? o.reason.trim() : '';
  if (o.violation === 1 && !reason) return null;             // cross-field rule
  return { violation: o.violation, reason };
}

Two properties of that code are worth stating plainly. First, a reply that quotes a message containing a brace and then gives its verdict parses correctly, because the scan starts at the end. A first-to-last slice would fail on the same reply. Second, the extractor is itself a loop over attacker-influenced text, so it needs the tests any hostile-input parser needs, and a clean verdict is the least informative one. Give it replies that begin with an opening brace and contain a later closing one without ever parsing, replies that are nothing but a brace, a valid object followed by a stray brace, two objects, and the empty string, and assert that each returns or fails promptly. The explicit guards in the loop exist because lastIndexOf treats a negative position as 0, so stepping back one index from index 0 does not end the scan.

Validate: check the claim against the source

For a model that justifies each judgment with a quote, the contract has a deterministic half. The quote either appears in the text it claims to quote or it does not, and no model is needed to decide that:

const norm = (s: string) => s.toLowerCase().replace(/\s+/g, ' ').trim();

// `shown` is the exact string the model was given, after any sanitizing.
if (claim.hit && !norm(shown).includes(norm(claim.evidence))) {
  return { error: 'evidence is not a quote from the source' };   // contract violation, not a miss
}

The comment on shown carries the subtle part. If you sanitize the input before the model sees it (stripping invisible characters, normalizing Unicode), the model quotes the sanitized text. Compare against the unsanitized original and an honest answer fails the check for a reason nobody can see. Apply the same transforms on both sides, and put a test on that seam.

Trade-offs

• Strictness converts malformed replies into the no-verdict state, and that state has a price. The tighter the contract, the more replies land in it. If the policy is a retry, the worst case per request is two model calls, and your spend ceiling has to be sized for two. Measure the rate before you tighten.
• The right-to-left scan can try many slices. On a reply with many braces it makes many JSON.parse attempts, each over a long slice. In a quick test of the same scan without the attempt cap, a reply of a thousand opening braces followed by a thousand closing ones took more than ten seconds, so an output-token cap bounds the damage only loosely. That is why the sketch, and our shipped extractor with its cap of 64, count attempts and treat hitting the cap as a no-verdict.
• Schema mode is not free. On Anthropic's API a new schema costs a compile on first use and stays warm for a day after its last use, and a changed schema structure starts that clock over. Swapping the output_config.format parameter mid-conversation also drops that conversation's prompt cache. Be careful what you put in the schema, too: a field that asks the model to show its reasoning can draw a refusal, and a request for a brief explanation is the documented way around it.
• A source-relative check rejects honest paraphrase. Requiring a verbatim quote makes the grader harder to fool and easier to trip. Normalizing whitespace, case and quote style recovers most honest cases. It never recovers all of them, so the fallback path has to be one you are happy for real users to land on.
• Every no-verdict policy has a failure mode. Fail closed turns a provider outage into an outage for your users. Fail open turns it into an unscreened request. Degrade-and-record-nothing keeps the data clean and costs the user a step. Pick the one whose failure you can afford, and write the reason where the next reader will find it.

When not to use this

Skip the validation layer when no code acts on the reply: a draft a human reads and edits needs a quality bar, not a parser. Skip most of it when the contract really is one closed field, a label from a short list, the consumer tolerates a default when the label is missing, and the vendor's constrained mode covers your model. There the schema is enough, and your remaining job is to check stop_reason and handle the refusal and the truncation. Do not import this pattern into tool calls the agent makes against your systems; that boundary has its own controls (see human-in-the-loop approval and the capability latch), and a well-formed tool call can still be the wrong call. And if a wrong verdict is cheap and reversible, with nothing downstream that reads its absence, a counter and a log line are the most that is worth building.

As-built evidence

aiArch runs three model-verdict parsers in src/lib/, written for different consumers; this page walks through two. The third, for practical-task grading in practicalGrade.ts, uses the same first-brace extractor as the short-answer grader. None uses a vendor schema mode today: each asks for JSON in the prompt and validates in code. Neither our decision log nor the input gate's design spec records why, so read that as the state of the code, not as a recommendation. Nothing in src/ sets a vendor output schema, and the only tool-choice field the model wrapper sends is auto (src/lib/llm.ts:429).

The input gate. extractLastJsonObject and parseJudgeOutput (src/lib/injectionJudge.ts:86-127) implement the right-to-left scan and the contract: violation must be exactly 0 or 1, and a reason is required when it is 1. The no-verdict state is a first-class member of the result type, unavailable, alongside legitimate and attempt_override (:174-177). An unparseable reply returns unavailable, still records the usage that was billed, and counts toward the circuit breaker (:209-216). The consumer then does what the layered-defense page argues for on purpose: unavailable falls through and the turn proceeds, fail-open, because the gate is one layer among deterministic controls that hold without it (src/lib/coach.ts:411). Regression tests for the parser are in test/injectionJudge.test.ts:59-203, including a preamble that contains a brace and a leaked reasoning channel (:89-105), and the mapping from an unparseable reply to unavailable with usage is at :241-246. Writing this page found a defect in that extractor: both loops stepped back with lastIndexOf(..., i - 1), which never ends once i reaches 0, so a reply whose scan walked back to a brace at index 0 without a parse (an unparseable {...} or a } at index 0) spun synchronously, and the 8-second abort on the call cannot interrupt a loop that never yields. It cost availability and CPU, not the tutor's safety: a killed request never reaches the tutor. The fix guards both steps (:108-111), adds a cap of 64 JSON.parse calls (:83) whose exhaustion returns no verdict, and tries one parse first from the last {"violation": to the last closing brace, so braces echoed inside the verdict's own reason cost nothing as long as the verdict opens with its violation key and no stray closing brace follows it. The regression tests had no reply that reached index 0 without a parse; they do now (:114-119), and the anchored parse and the cap have their own tests (:143-146, :171-182). Some replies still exhaust the cap. The anchored parse helps only when the verdict opens with its violation key, as the prompt asks, and nothing after it adds a closing brace or another {"violation":. Otherwise the scan pays a parse per opening brace before each closing brace it tries, so 63 or more stray closing braces after the verdict, or 64 or more braces inside a reason that comes before the violation key, exhaust the cap and fail open, and braces before the verdict lower both counts. The gate is skipped and the turn proceeds, as for any no-verdict. Tests pin the first case and none covers the second: one pins the null result for a noise-then-stray-brace reply (:148-150) and another pins the 64-call stop for 200 trailing braces (:173). The story of how the original parser's weakness was found, and why a severity rating on it was wrong, is not repeated here: it lives in the review workflow.

The short-answer grader. parseShortAnswerJudge (src/lib/grade.ts:395-429) takes the reply through the simpler extractor at :360-369, the span from the first brace to the last, and returns {error} for most contract breaks: not JSON, no results list, a rubric item the model omitted, or a claimed hit whose supporting quote is not in the learner's answer once both sides are normalized the same way the prompt normalized the answer (:374-385). A hit with an empty quote is downgraded to a miss rather than rejected. We do not publish the grading prompt or rubric, only this parsing contract.

What the caller does with {error} is the part worth reading, and it is from the code, not from the comments. gradeShortAnswer calls the judge, and on {error} calls it once more (src/lib/grade.ts:591-595). A thrown error, a spent budget or a guardrail block short-circuits to the same fallback (:596-602). If the second reply is still an error the function returns the self-grade prompt (:603): correct is null, the mode is self_grade, and nothing is written to the attempts table until the learner submits their own grade (:558-566). No unparseable reply, and no reply that omits a rubric item, becomes a grade. One coercion remains, and it is worth flagging: a hit that is anything other than the boolean true (absent, a string, a number), or a quote that is empty or not a string, is scored as a miss rather than rejected (:418,421,425), so a wrong-typed reply can record a fail. The route's cost comment sizes the global free-plan daily ceiling for the two-call worst case (src/routes/shortAnswer.ts:33-34).

The same code shows two gaps, stated here because the pattern asks you to count the no-verdict state. First, nothing counts it: the {error} branch writes no log line and no counter, so the rate at which honest answers land on self-grade is not observable. Second, the parser has tests and the offline degraded path has tests (test/short-answer.test.ts:207-329, :420-427), and the offline eval harness mirrors the re-ask (src/evals/grader.eval.ts:157-170, test/grader.eval.test.ts:59), but no test drives the live-judge branch of gradeShortAnswer through re-ask and degrade end to end. That behavior is established by reading the code, not by a test that would catch a regression.

The vendor side of this pattern was also wrong in our own curriculum, twice. The Track C lesson on the Messages API and structured output taught a schema-keyword table that had drifted from the live documentation and understated the failure, since an unsupported keyword is a 400 rather than being ignored (Q-537, corrected 2026-08-17). It also presented forced tool_choice as the portable route to structured JSON, which now returns a 400 on four current models (Q-746, corrected 2026-10-19). Both are the same lesson as the rest of this page: the vendor's contract is an input to your design, it moves, and it is checked at request time, not when you write the code.

Changelog

• 2026-10-19 — Initial publication. Vendor facts (Anthropic structured outputs and forced tool use, OpenAI Structured Outputs) verified against the live documentation on this date; as-built lines re-read at the repository HEAD of the same day.