Pattern · last reviewed 2026-08-25

Pinning a model is not pinning its behaviour

Context & problem

A team pins a model id — claude-opus-4-8, say — because "pinned" reads as "stable." It is stable in exactly one respect: Anthropic's own deprecation policy says an active model id will not be retired without at least 60 days' notice, delivered by email and in the documentation. That is a real guarantee and it is worth having. It is also not the guarantee most teams think they bought.

The id names a model. It does not freeze the API surface around that id, the infrastructure serving it, or the numbers a tokenizer assigns to your prompt — and all three can move while the id you pinned sits unchanged in your config. Anthropic's own parameter table makes this concrete: as of the model-deprecations page fetched for this pattern, temperature, top_p, and top_k are deprecated on Claude Opus 4.7 and later and now return a 400 error when set to a non-default value — on ids that were already active, that nobody retired, that a pinned config still names correctly. A team that pins the id and calls the job done has pinned against the wrong failure mode.

Forces

• What retirement covers vs what it doesn't. Anthropic's lifecycle terms — Active, Legacy, Deprecated, Retired — and its 60-day notice period are about the id disappearing. They say nothing about the id's behaviour while it remains Active, because that was never the promise.
• A default isn't visible in the id string. Anthropic's own docs say effort defaults to high on Sonnet 5 and Opus 5 — on every route, first-party API included — but a pinned id doesn't print that, and it's easy to assume a default you first noticed on one route is specific to that route. Our own guard was scoped that way at first and had to be widened; detailed below.
• Stability vs staying current. An indirection layer plus an eval suite is a standing cost that buys you a warning, not a guarantee; skip it and you find out from a user complaint instead of a red test.
• What you can verify vs what you can only hope. Some mitigations are provable with a live call; others — like whether a wire-level flag actually does what its name says — stay unverified until you spend to check, and the honest version of this pattern says so rather than claiming a fix that was never confirmed.

The pattern

Treat "what model does this call use" as two separate questions with two separate answers. The first is identity: a stable role→id map, one function, so a model swap is a one-line edit (this is the model router pattern). The second is behaviour: does the id still do, today, what your product depends on it doing. The router answers the first question. Only an eval answers the second, and an eval only answers it if it actually runs before a ship — as the release gate the eval-harness gate pattern already describes, pointed here at model drift rather than at your own prompt changes.

Between the two, put an indirection boundary: the canonical id your code reasons about should not be the literal string sent on the wire. A provider slug, a gateway route, or a partner-platform mapping can all drift independently of the model itself, and when one of them does, you want that to be a one-file diff, not a grep across every call site.

Reference implementation notes

Anthropic

Anthropic's own model-lifecycle documentation is explicit about what a pinned id buys you: four states — Active, Legacy, Deprecated, Retired — and "Anthropic notifies customers with active deployments for models with upcoming retirements, providing at least 60 days' notice before model retirement for publicly released models." That is a real, citable guarantee against the id disappearing out from under you.

It is a narrower guarantee than it sounds, because the same page documents behaviour changing on an id that stays Active. Its parameter-deprecation table states: "temperature, top_p, top_k — Deprecated (Claude Opus 4.7 and later) — Returns a 400 error when set to a non-default value on Claude 4.7 and later models" — with the recommended fix being to drop the parameter and steer behaviour through prompting instead. Separately, the same page notes the Python SDK (v1.0 and later) removes those three parameters from the request types entirely, so old code that sets them raises a client-side TypeError before the request is even sent. Neither event retires a model id. Both change what code written against that id needs to do.

AWS (Bedrock)

Amazon Bedrock runs its own, separately-dated lifecycle on top of Anthropic's: a model on Bedrock is Active, Legacy, or End-of-Life, and — this is the guarantee closest to what teams assume "pinned" means — "once a model launches on Amazon Bedrock, it will remain on Amazon Bedrock for at least 12 months before the EOL date," with a further minimum of 6 months in the Legacy state before EOL, per Bedrock's model-lifecycle documentation. The same page is explicit that this is a Bedrock-specific clock: "Model lifecycle dates on this page are specific to Amazon Bedrock and may differ from dates published by model providers (such as Anthropic or Cohere). For Amazon Bedrock usage, only the dates on this page apply." A team running the same nominal model through both Anthropic's API and Bedrock is tracking two independent retirement calendars under one id, and the Bedrock guarantee — like Anthropic's — covers the id's continued existence, not what it does while it exists.

Trade-offs

• The eval suite is the actual cost. An indirection layer is cheap; a scored eval suite that would actually catch a behaviour drift is not — it needs real cases, a scoring function, and someone reading the score when it moves. Skipping the eval and keeping only the indirection buys you the wrong half of the pattern.
• A stale eval suite is worse than none — it launders drift as "unchanged." If the suite encodes last quarter's answers as ground truth, a model that got worse in a way the suite doesn't test still shows green.
• Two independent lifecycle clocks if you run on more than one platform. Anthropic's own retirement schedule and a partner platform's (Bedrock's, Vertex's) can diverge for the same nominal model — verify against the platform you actually call, not the vendor's headline page.
• A wire-level mitigation you have not tested is not a mitigation. Sending a flag that is supposed to suppress a behaviour is not the same claim as having confirmed it suppresses the behaviour — see the as-built section below for a case where we shipped the first without the second, on purpose, and said so.

When not to use this

Skip it early, on a product still finding its shape. Pinning plus an indirection layer plus a shadow-eval suite is a standing cost, and it buys a stability you may not want yet: pin at the wrong moment and you sit out a generation of cheaper, better models while your eval suite encodes last quarter's answers as ground truth. If you ship weekly and would take a free quality upgrade on sight the moment it's available, float the version and spend the effort on the harness instead — the harness is what makes the pin optional later, and it is the piece worth building first regardless of which way you land on pinning itself.

As-built evidence

aiArch pins by role, not by literal wire string. modelFor() in src/lib/llm.ts returns a stable canonical id per role (hint, tutor, eval, grader, safeguard); a separate wireModel() maps that canonical id to the OpenRouter slug actually sent on the wire, through a small lookup table the code comments explicitly mark "re-confirm on release." The two-function split exists because the mapping has already drifted once in production: Sonnet 5 replaced Sonnet 4.6 as the tutor role's id on an unchanged request shape, and the file keeps the retired slug mapped for historical cost accounting rather than deleting it outright.

The behaviour-drift half of this pattern is not hypothetical here either. Our coach began returning empty replies that still billed, with no id change and no code change: claude-sonnet-5 ships with adaptive thinking on by default, and reasoning tokens count against max_tokens — a hard-thinking turn burned the whole output budget with zero visible content, a successful call that said nothing. We first hit this via OpenRouter serving the id through Amazon Bedrock (2026-07-08), and the fix we shipped was scoped to that route, reading at the time as a Bedrock-specific workaround. It wasn't: Anthropic's own docs state effort defaults to high on the first-party Claude API too, for both Sonnet 5 and Opus 5, so the guard was widened (2026-07-26) to run on every Anthropic route the code sends, not only the one where it was first caught — a narrower fix would have quietly regressed the moment traffic moved off Bedrock. We now send reasoning: { effort: 'none' } on every Anthropic slug (buildChatBody, src/lib/llm.ts:359-390), and the honest status of that fix is recorded, not assumed: test/llm.test.ts asserts the wire shape under the title "sends reasoning effort none on anthropic slugs (wire shape, not efficacy)" — deliberately not claiming the parameter works, because whether it actually suppresses thinking on the routed models is still an open, unsettled question in our own tracker (Q-353). Settling it takes one live gateway call reading the thinking-token count back off the usage object, which is real spend, and we have not spent it. What we shipped instead as the control that actually holds is runCoachLoop yielding an in-band "ran out of reply budget" message on any turn that produced no visible text (src/lib/coach.ts) — visible and survivable whether or not the wire parameter does anything at all. That ordering — verify what you can, and make the thing you can verify the one holding the weight — is the transferable part.

The eval harness half of the pattern runs as npm run eval (test/coach.eval.test.ts, test/grader.eval.test.ts, test/practical.eval.test.ts), scored suites gating the same release path a prompt change would go through — the mechanism the eval-harness gate pattern names generally, pointed here at "did the model change under me" rather than only "did my prompt change break it." It is a role-scoped harness for our own call sites, not a general Anthropic-fleet drift monitor — the 2026-08-24 currency-watch sweep that found the temperature/top_p/top_k deprecation live on our own tracked ids (Opus and Sonnet tiers, unretired, unchanged) is what a currency-watch process catches; the eval suite is what would catch a regression in what those ids actually produce.

Changelog

• 2026-08-25 — Initial publication. Anthropic model-deprecation policy and parameter-deprecation table verified against the live docs; AWS Bedrock model-lifecycle policy verified against the live user guide.