Pattern · last reviewed 2026-07-26

Role-based model routing: stop paying Opus prices for Haiku work

Context & problem

An LLM feature ships against one model. It works, so every call in the system — the one-line classification, the multi-turn tutoring, the end-of-course judgment — goes to that one model. If it is the flagship, you pay flagship rates for work a model a fifth the price does identically well; on Anthropic's current rates that is $5/$25 per million tokens for Opus 4.8 where Haiku 4.5 bills $1/$5. If it is the cheap model, the hard calls quietly underperform and nobody connects the dropped-quality complaints back to model choice.

The reflexive fix is a classifier router: before each request, call a small model (or a heuristic) to score difficulty, then dispatch to a tier. It sounds right and it is mostly wrong for application code. You have added a call, a point of latency, and a second failure mode to every request, to recover a signal you usually already have. In an application you wrote, you know at the call site what kind of work this is — a hint is a hint, a rubric grade is a rubric grade. The task class is the routing key. The difficulty guess is only worth buying when the workload is genuinely opaque, and even then it belongs to the platform, not your request path.

Forces

• Cost vs quality. The cheapest model that clears the bar for a role is the right model for that role — but "clears the bar" is per-role, and a global default optimizes neither end.
• Simplicity vs adaptivity. A static role→model map is trivial to reason about and test; a per-request classifier adapts to inputs but adds a call, latency, and a way to be wrong. Most application traffic does not need the adaptivity.
• Isolation vs sprawl. Model ids, provider slugs, and prices churn constantly. Left inline they leak into dozens of call sites; a breaking change becomes a scavenger hunt.
• Spend ceiling vs availability. A hard budget cap protects the invoice but can hand a user an error mid-task; a soft downgrade keeps the feature alive at lower quality. You want both, on different thresholds.

The pattern

Name your roles — the small, closed set of task classes your product actually makes. Map each role to a model tier in one place. Every call site names a role, never a model. That single function is the only code that knows a model id, which makes model churn a one-line edit and cost a property of the role rather than an accident of where the call was written.

Two things ride alongside the map. First, prompt caching: mark the stable prefix — the system prompt and tool definitions, which are identical across calls in a role — as cacheable, so each call re-reads that prefix at roughly a tenth of the input price instead of paying full freight every turn. Second, fallback routes: on budget pressure, downgrade rather than fail — a soft cap sends traffic to the cheap tier; only a hard cap returns an error, and that error is a graceful in-band message, not a stack trace.

Reference implementation notes

Anthropic

The tiers are the model family: Haiku for the cheap role, Sonnet for the default, Opus for judgment. The router is a plain switch — no framework earns its place here:

// one function knows model ids; every call site names a role
type Role = 'hint' | 'tutor' | 'grader' | 'eval';

function modelFor(role: Role): string {
  switch (role) {
    case 'hint':   return 'claude-haiku-4-5';   // $1 / $5  per MTok
    case 'tutor':  return 'claude-sonnet-5';    // $2 / $10 per MTok
    case 'grader': return 'claude-sonnet-5';
    case 'eval':   return 'claude-opus-5';      // $5 / $25 per MTok
  }
}

Prompt caching is the multiplier on top. Mark the system prompt and tool definitions with a cache breakpoint; a cached prefix read bills at roughly 10% of the input rate, so a long stable system prompt stops being a per-turn tax. Anthropic's default cache lives ~5 minutes with a 1-hour option, and batch submission halves the rate again for anything that need not be live. The one trap that survives a model swap: tokenizers drift. Sonnet 5 renders the same text as roughly 30% more tokens than the prior generation, so a cost estimate baselined on an old tokenizer silently under-reads after an upgrade — re-measure token counts, do not assume them.

AWS (Bedrock)

When the workload genuinely is opaque — you cannot name the role at the call site — this is exactly where a difficulty router earns its place, and AWS ships it as a managed service so you do not build one. Amazon Bedrock Intelligent Prompt Routing (generally available) predicts, per request, which model in a family will meet the quality bar at the lowest cost and dispatches accordingly; AWS reports up to ~30% cost reduction on suitable workloads. Two constraints shape when to use it: it routes within a single model family, and it is optimized for English prompts. Combine it with Bedrock prompt caching for the same stable-prefix saving. Use the managed router for opaque traffic; keep the static role map for the calls whose class you already know — they are not mutually exclusive.

Cloudflare (AI Gateway)

AI Gateway is the operational half: it sits in front of the provider and gives routing, caching, and spend control without touching call-site code. Dynamic routing composes a flow (visual or JSON) that evaluates conditions, enforces quotas, and picks a model with fallbacks — a primary node whose fallback points at a second model, on a different provider if you like, that fires when the primary errors or times out. Response caching serves identical requests from Cloudflare's edge, cutting latency and repeat-call cost, with per-request TTL control. Spend rules live in the dashboard: a monthly cap can be configured to return 429 so the app can degrade in-band; a soft rule can dynamically route down to the cheap tier. This is the gateway layer doing budget-pressure fallback for you, above whatever static map your code already carries. (Note: the older Universal Endpoint is deprecated in favor of dynamic routing — build new flows there.)

Trade-offs

• The map needs maintenance. A static role→model map is only as good as its last review. New roles get bolted onto an existing tier out of laziness; model upgrades change the price/quality frontier. Revisit the map on every model release, not never.
• Roles can multiply. The pattern is clean with a handful of roles and muddy with thirty. If you find yourself minting a role per call site, the abstraction has inverted — collapse them back to task classes.
• Caching adds coupling. Prompt caching only pays when the prefix is genuinely stable; interleave volatile data into the cached region and you thrash the cache and pay the write premium for nothing. It constrains how you order a prompt.
• Soft downgrade hides regressions. A fallback to the cheap tier keeps the feature alive but silently lowers quality; without a per-call cost-and-model log you will not notice you have been serving the budget tier for a week.

When not to use this

Skip it when there is genuinely one kind of call. A single-purpose feature — one classification endpoint, one summarizer — has one role, and "routing" over one role is a switch with one arm. Pick the right model once and move on; the map is overhead with nothing to organize.

Skip the static map, specifically, when you cannot name the task class at the call site — open-ended user prompts of unknown difficulty. That is the case a difficulty classifier or a managed router (Bedrock Intelligent Prompt Routing) is built for; forcing a hardcoded role map onto opaque traffic just moves the wrong guess into your code. And do not reach for either flavor of routing to rescue a quality problem that is really a prompt or eval problem — if the top tier does not clear the bar, a router will not, and you are tuning the wrong knob. Fix the prompt and the eval first (see the eval-harness gate), then route.

As-built evidence

aiArch runs this pattern in production. The coach and assessment features route by role through one function — modelFor() in src/lib/llm.ts — with hint on Claude Haiku 4.5, tutor and grader on Claude Sonnet 5, and safeguard — the coach's input-injection judge — on the open-weight gpt-oss-safeguard-20b rather than an Anthropic tier, which is the case for a role map rather than a family ladder. A fifth role, eval (Claude Opus 5), is declared and reserved for capstone evaluation but has no live call site yet. Call sites name a role and never a model, so the tutor role's 4.6→5 upgrade was a one-line change in that file and nothing else. The same file marks two prompt-cache breakpoints — the stable system-plus-tools prefix and a rolling conversation prefix — so multi-turn coaching re-reads its transcript at cache-read prices; src/lib/cost.ts bills cache reads at 0.1× input and logs a per-turn USD figure per route, which is how a silent downgrade would surface.

The whole stack sits behind Cloudflare AI Gateway on the authenticated path, and the budget-pressure path is handled in code from the gateway's response through to each caller's degradation. Cloudflare answers an exceeded spend limit with 429 Too Many Requests, and src/lib/llm.ts maps any 429 on the gateway path to a BudgetExhaustedError — any, deliberately, because a spend-limit 429 carries no header or error code distinguishing it from a rate-limit one. All six live callers then degrade instead of throwing: the coach yields an in-band usage-limit turn, short-answer grading falls back to learner self-grade, practical grading returns needs_review over a deterministic verdict that already stands, the guide coach serves the grounded answer it had already assembled, the injection judge fails open, and the Brief drafter skips its remaining sources. That handler is the part we can show you. Which spend-limit rules are configured on our gateway is dashboard state, not repo state, so this page claims no specific rule — neither a monthly ceiling nor the cheaper-model fallback (a dynamic route whose primary model carries the limit). Both are gateway capabilities this architecture is built to absorb; the code path that survives them is what the as-built evidence covers. The routing indirection is deliberate churn-isolation: provider slugs, gateway wiring, and the cache markers all live in the one src/lib/llm.ts boundary, so a breaking upstream change stays a one-file fix. Cost discipline here is not incidental — it is the same Track 0 and Track B lesson the platform teaches, applied to itself.

Two further things about the map, both bought with our own spend. The first is how a role gets its model: by running it, not by reading about it. The default image model for our graphics pipeline was picked at design time from capability research, written into the decision record as ratified, and flipped the same day. A three-candidate bake-off on the first real asset — identical prompt, both models — put openai/gpt-5.4-image-2 visibly closer to our brand palette than FLUX.2 Pro, with finer linework, and cheaper per image at $0.056 against $0.075 actual API-reported. The default changed and the loser stayed reachable behind the script's --model flag for the next comparison. Total cost of the exercise, including overturning the decision it had just ratified: $0.393. That figure is the whole argument. A role map is a set of empirical claims about which model clears which bar, and the price of testing one of those claims is rounding error next to a month of quietly running the wrong model. This was an image role rather than a text tier, but the discipline carries: the review cadence in the trade-offs above is worth very little if the review is a reading exercise.

The second is a warning about what the map actually pins. It pins a model id. It does not pin the stack serving that id. Our coach started returning empty replies that still billed, with no code change and no id change: OpenRouter had begun serving anthropic/claude-sonnet-5 through Amazon Bedrock, which serves it with extended thinking enabled by default, and reasoning tokens count against max_tokens. The gateway log for a failing turn showed exactly 1024 output tokens — our default ceiling — a body full of reasoning blobs, zero content, finish_reason: length, and no error anywhere in the chain. The model had spent its entire output budget thinking and had nothing left to say it with, and the whole thing arrived as a successful call. A router that maps roles to ids inherits, silently, every routing decision the provider makes upstream of it.

We shipped two responses to that, and the honest report is that only one of them is proven. buildChatBody in src/lib/llm.ts now sends reasoning: { effort: 'none' } on every Anthropic slug — and whether that parameter actually suppresses thinking on the models we route is an open question in our own tracker, not a settled result. What we established is that the gateway accepts the field without rejecting the call, which is a weaker claim than it looks: the reading in which it is a silent no-op, leaving the provider's own default in force, has not been ruled out. Settling it takes one live call reading the thinking-token count back off the usage object, which is real spend against a live model, and we have not spent it. The response carrying the actual weight is the other one: runCoachLoop yields an in-band "ran out of reply budget" message on any final turn that produced no visible text (src/lib/coach.ts), so the failure is visible and survivable whether or not the wire parameter does anything at all. That ordering is the transferable part. When you mitigate provider drift, make the control you can verify the one holding the weight and treat the wire parameter as the optimisation — otherwise your defence against an invisible failure is itself unverified, which is the same problem one layer up.

Changelog

• 2026-07-13 — Initial publication. Pricing and provider-feature state verified against current Anthropic, AWS, and Cloudflare docs.
• 2026-07-25 — Updated the reference implementation's eval role from Claude Opus 4.8 to Claude Opus 5, matching the site's own modelFor() upgrade (same $5/$25 rate).
• 2026-07-26 — Corrected the as-built section: the eval role is declared but has no live call site, and the live fourth role is safeguard (the coach's input-injection judge, on an open-weight classifier).
• 2026-07-26 — Second as-built correction: this page asserted a configured hard monthly cap and a soft rule downgrading to Haiku. Which spend-limit rules exist on our gateway is dashboard state, not something the repo can evidence, so the as-built claim is now the 429 → BudgetExhaustedError handler and the six callers that degrade on it; the rules themselves are described as gateway capabilities.
• 2026-08-10 — Added two as-built passages. A $0.393 three-candidate bake-off that flipped a model default on the day it was ratified, as the argument for choosing a role's model empirically. And provider drift: a role map pins a model id, not the stack serving it — with the extended-thinking failure that produced empty billed replies, and an explicit statement that our wire-level mitigation is unproven while the in-band handler beneath it is what actually holds.
• 2026-08-15 — Named the LLM router in the opening sentence; the pattern name, id and URL are unchanged.