Pattern · last reviewed 2026-10-19

An LLM Error Means What Your Path Says It Means

Context & problem

The standard advice is a single sentence: retry 429 and 5xx with backoff, do not retry the other 4xx. It is correct for one client talking to one provider through that provider's SDK. It stops being correct the moment a layer you do not control starts producing status codes of its own, which is what an AI gateway, a model router or a billing proxy does. The sentence assumes the number describes the provider. Behind a gateway it may describe your budget, your credential, or a routing decision, and for those a retry is either useless or harmful.

The harm is asymmetric. Retry a spend-limit rejection and you hammer a limit that will not lift until a window resets. Treat a transient provider blip as a hard stop and a learner sees an error that a second attempt would have cleared. Neither mistake raises an alert. Both look like normal 4xx or 5xx traffic in a dashboard, which is why the classification has to be written down in code, once, before the retry loop exists.

Forces

The vendors do not agree with each other, and several of them reuse a code for something you would not guess. Four facts are worth knowing before you write a single branch. Each is stated here as the vendor documents it.

• A 400 can be money. Anthropic returns a 400 invalid_request_error, not a 429, when you hit a spend limit you configured on an organisation or workspace. The Claude Code workspace is the documented exception and can return a 429. A rule of "never retry a 400" is right here, but a rule of "a 400 means my request is malformed" sends you debugging a payload that is fine.
• A 429 can be a wall, not a wait. The same page documents 429 rate_limit_error for a rate limit and for a usage tier's monthly spend cap, and says a spend-cap 429 "has no retry-after header and keeps failing until access resumes". The rate-limits page gives the discriminator: on the Messages API a spend-cap 429 carries error.details.error_code set to enforced_spend_limit_reached, and the vendor names that field as the way to separate a spend-cap rejection from a throttle. The missing header is supporting evidence only, since that page also says not every 429 carries retry-after. A client that retries every 429 on a timer is right for one and wrong for the other, and the vendor's own SDK retries cannot clear the spend-cap case.
• A 503 can mean two unrelated things in two layers. OpenRouter documents 503 as "no available model provider that meets your routing requirements", with a Retry-After header that may be present, which reads as a routing condition that can clear. Cloudflare's AI Gateway changelog for 6 October 2026 documents that when Unified Billing is in use and the provider rejects the credentials, the response is now HTTP 503, while other setups return 401 with error code 2009. Cloudflare's entry scopes this to POST /ai/run and does not say whether a client should retry. A 503 is conventionally read as transient; that is my inference, not Cloudflare's guidance, but it is the reading a generic retry policy will apply. Before the change, a rejected Google Vertex credential came back as a 500 that the gateway retried upstream, which is the same mistake made one layer down: a failure that cannot succeed, handled as one that might.
• A failure can arrive as a 200. Both Anthropic and OpenRouter document that a streamed response can fail after the 200 has been sent, and that the status line is already final by then, so the error has to travel as an event in the stream body. OpenRouter's form is an event whose finish_reason is error. A classifier that only reads statuses is blind to this entire class.

Two more forces come from the layers the status sits in. The gateway's own spend-limit rejection is a bare 429 with no documented distinguishing header or code, and its guardrail block is identified by an error code (2016 for a prompt, 2017 for a response) that Cloudflare documents without naming an HTTP status. Those are the two classes this site has already written up in depth, so this page links them instead of repeating them: the 429 in enforceable spend ceilings, the guardrail block in defense in depth, and the role map that decides which model the call goes to in role-based model routing.

The pattern

Write the classifier as a pure function from what you observed to a disposition, and make the retry loop a consumer of it, not the other way round.

• Key on three things. The path (which layers did this call cross: direct to a provider, through a router, through a gateway), the status, and, only where the vendor gives you nothing better, a narrow match on the body. Where the vendor documents a structured field, as Anthropic does with error.details.error_code on its spend-cap 429, prefer it to free text in the message.
• Four dispositions, no others.
  • Retry unchanged: the failure says nothing about your request or your account and a second attempt has an independent chance. Transient provider errors and overload belong here, bounded and with backoff.
  • Retry changed: the failure names something specific you can remove or alter, and the changed request is the one you send. It is a retry of a different request, so it needs its own bound of one.
  • Degrade: the call cannot succeed now and the cause is not your request: a budget, a limit, a guardrail block. Retrying cannot help and costs something. Give the user a path that works without the model, and say something true about why.
  • Surface: your request is wrong, the credential is rejected, or you do not know. Fail loudly and early, with a message that carries the status and the vendor's code but not the body.
• Unknown is surface. A default of "retry" turns every misclassification into extra traffic against a limit. A default of "surface" turns it into one visible error that someone reads.
• Run the classifier on the final response. If one class retries with a change, classify the second response too. The retried call can be the one that hits the budget.

A minimal sketch

The shape, generically. This is a sketch of the rule, not our code; it shows an ordered list with a default at the bottom, and the one place a retry changes the request.

// Pure: no I/O, so every row of the table is one unit test.
type Disposition = 'retry' | 'retry-changed' | 'degrade' | 'surface';

function classify(path: 'gateway' | 'direct', status: number, body: string): Disposition {
  if (status === 429 && path === 'gateway') return 'degrade';          // no discriminator exists, so the status is the signal
  if (status === 429 && /spend|budget/i.test(body)) return 'degrade';
  if (status === 424 && /\b201[67]\b/.test(body)) return 'degrade';
  if (status === 400 && /cache/i.test(body)) return 'retry-changed'; // resend WITHOUT the cache markers, at most once
  return 'surface';                                                  // unknown never retries
}

A caller that talks straight to one provider and can read retry-after should add a row that waits and retries on a 5xx or an overload response, bounded. That row is the only one of the four dispositions that is safe to add without reading the body, and it is the row most pages on this topic already cover. The rest of the table is the part they leave out.

Trade-offs

• Body matching is brittle by construction. The cache row keys on a substring and the guardrail row on a code or a phrase. A vendor who rewords a message silently moves a failure into the "surface" default, which is the safe direction but still a regression. Pin each row with a test that feeds the exact body shape, and treat a changed vendor message as a breaking change.
• "Surface" costs availability. A client that retries nothing but one class shows a transient provider error to a person who would have been served by a second attempt. That is a real price, and we pay it: our client has no overload retry, and we have not measured how often a learner meets that; a bounded retry on overload is the obvious next row, and the reason to add it is a count, not a feeling.
• Hedged messages are the price of an ambiguous signal. When a class cannot be told from its neighbours, the user-facing text has to be true under every cause. That is less satisfying than naming the reason, and it is correct.
• Retrying after a timeout can pay twice. An aborted generation may still run, and bill, upstream. A timeout is therefore a "surface", not a retry, unless you can show the first attempt was cancelled.
• The retry-changed row is a second request you wrote. It doubles the failure surface for that class and needs its own observability. A warn line with no counter, which is what we have, tells you it happened and not how often.

When not to use this

Skip the classifier when you call one provider directly through its official SDK. Anthropic's official SDKs already retry the transient classes themselves, two retries by default with backoff, reading retry-after when it is present, and max_retries changes that. The vendor has done the classification for the layer it owns, with one documented exception, the spend-cap 429, which no retry can clear; set that option to what you want and move on. This pattern starts to pay at the first layer you add that the SDK does not know about.

Skip it for offline batch work where nobody is waiting and a uniform retry-with-backoff over everything is cheaper than being right, provided a spend ceiling you can read sits underneath it. And do not build the table for a call path that has one failure mode you handle identically, such as a best-effort enrichment you drop on any error. Classification earns its place when two failures want different responses.

How to test for it

Test the table, not the loop. For each row, feed the classifier the exact status and body shape and assert the disposition. Then add the cases that guard the boundaries: an unrelated 400 must not retry, an unrelated 424 must not be read as a guardrail block, and a 429 on the gateway path must not depend on its body text. The last of those is the one that fails quietly in production, because a differently worded rejection passes every test that only exercises the wording you expected.

As-built evidence

aiArch's model client is src/lib/llm.ts, reaching OpenRouter either directly or through an authenticated Cloudflare AI Gateway. The classification lives in one function, postChat (src/lib/llm.ts:510-563), with the thrown surface in providerError (llm.ts:574-582), and the unit tests for the 400, 429, 424 and generic-error rows are in test/llm.test.ts (the 400 cases from line 530, the gateway 429 cases from line 591, the 424 cases from line 628). The per-attempt timeout bound has no unit test of its own; the only abort test there covers a caller-supplied signal (line 660). The table above is those two, row for row, plus mapFinish for the mid-stream row.

There is exactly one retry. A 400 is retried once, without the cache markers, and only when the response body mentions cache (llm.ts:526-532). The reason is that those markers are an optimisation we added, so a router that starts rejecting them should cost us money, never availability. An unrelated 400 propagates at once, and a test pins that (test/llm.test.ts:561). The retried response goes through the same 429 and 424 checks as the first, because the second call is the one that can hit the budget.

The 429 and the 424 are mapped to typed errors, not retried: any 429 on the gateway path becomes a BudgetExhaustedError, a direct-path 429 only when its body says spend or budget, and a 424 carrying code 2016 or 2017 becomes a GuardrailBlockedError that records which side blocked (llm.ts:545-562). All six runtime callers degrade on any failure; the typed errors change what they say or do, not whether they degrade. The coach answers in band, with a limit-specific or filter-specific message for the typed errors and a generic try-again message for anything else (src/lib/coach.ts:546-585); short-answer grading falls back to the self-grade prompt either way (src/lib/grade.ts:596-602); practical grading keeps its deterministic verdict and marks the rubric pass for review, with a different note for a budget or guardrail block (src/lib/practicalGrade.ts:525-538); the vendor-watch drafter stops the run on a budget error and skips only the failing source otherwise (src/lib/drafter.ts:217-225); the injection judge fails open on anything (src/lib/injectionJudge.ts:191-239); and the public guide coach serves the grounded answer it had already assembled (src/routes/guideCoach.ts:184-196). The client classifies once and throws; each caller decides what degrading means, including for the failures the client left unclassified, so “surface” in the table means the client throws, not that a learner sees a raw error.

Two details carry weight that is easy to miss. First, every attempt gets a fresh 60-second timeout (llm.ts:195, 516-524), the retry included, because an upstream that accepts the connection and sends nothing would otherwise hold the coach's per-learner state open and block every later message. Second, a failure is thrown as the status plus the provider's numeric code and never the body (llm.ts:574-581). OpenRouter documents that a moderation error can carry an excerpt of the flagged input, limited to 100 characters, and our logs are a store neither the retention job nor the erasure routine reaches, so the body stays out of the message by design.

The judge shows the other half of the table: a classified failure can still be the wrong place to retry. The injection judge bounds its own wait at 8 seconds and opens a breaker after three consecutive failures for 60 seconds (injectionJudge.ts:135-168), then fails open. It calls once and does not retry at the HTTP level; the call sits on the critical path of every learner turn, so a retry would only lengthen the wait for a control that does not hold the weight. That fail-open posture is described in defense in depth. The guardrail side has its own direction. Cloudflare's usage-considerations page says that if any category is set to block and the evaluation is unavailable, the request is blocked, and that with flag-only settings it proceeds without evaluation; which of our categories is which is in our decision record.

There is no fallback model, on purpose, in one place. The safeguard role runs openai/gpt-oss-safeguard-20b, and Groq is the only endpoint serving it: we re-read OpenRouter's endpoint list for that slug on 2026-10-19 and it returns one entry, provider Groq (llm.ts:100-119). We looked for a same-contract alternative in July and found none. The judge needs a free-form policy prompt and a strict JSON verdict; the two nearby classifiers served on OpenRouter either take no policy or answer in a different format. The decision, logged in our tracker as Q-062 and closed on 2026-07-26, was a watch item, not a coded fallback: swapping a safety classifier changes the security posture, and a fallback that cannot meet the contract would be a control that looks present and does nothing. The cost of that decision: a Groq outage leaves the judge failing open for as long as it lasts. The breaker bounds the latency, not the exposure. Each isolate makes three failed calls, skips the judge for 60 seconds, then repeats, so learners do not pay the wait on every turn, and the control is inert either way.

What we do not do. We do not retry 5xx, overload or timeout responses; on the coach the learner is told to try again, and for a transient provider blip that is a worse experience than a bounded retry would be. We have no counter on the cache-strip retry, only a warn line. And we do not classify a mid-stream failure: mapFinish (llm.ts:438) passes an unrecognised finish reason through as the stop reason, so an OpenRouter error finish reaches the caller as a stop reason of error rather than a thrown error, a stream event with no choices is skipped (llm.ts:679), and complete() checks only the status (llm.ts:588), so the error body OpenRouter documents for a non-streaming request that fails after a 200 is unclassified too. I read the client and did not trace every caller's handling of that value, so treat the gap as open until someone does. The Unified Billing 503 does not apply to us as built: our calls use the gateway's OpenRouter provider passthrough with our own key, not POST /ai/run with Unified Billing, and I make no claim about what that path returns for a bad key.

Changelog

• 2026-10-19 — Initial publication. Anthropic, OpenRouter and Cloudflare AI Gateway error semantics read from the vendors' pages on this date; as-built claims verified against src/lib/llm.ts, its callers and test/llm.test.ts at the same time.