Pattern · last reviewed 2026-07-13

Role-based model routing: stop paying Opus prices for Haiku work

Context & problem

An LLM feature ships against one model. It works, so every call in the system — the one-line classification, the multi-turn tutoring, the end-of-course judgment — goes to that one model. If it is the flagship, you pay flagship rates for work a model a fifth the price does identically well; on Anthropic's current rates that is $5/$25 per million tokens for Opus 4.8 where Haiku 4.5 bills $1/$5. If it is the cheap model, the hard calls quietly underperform and nobody connects the dropped-quality complaints back to model choice.

The reflexive fix is a classifier router: before each request, call a small model (or a heuristic) to score difficulty, then dispatch to a tier. It sounds right and it is mostly wrong for application code. You have added a call, a point of latency, and a second failure mode to every request, to recover a signal you usually already have. In an application you wrote, you know at the call site what kind of work this is — a hint is a hint, a rubric grade is a rubric grade. The task class is the routing key. The difficulty guess is only worth buying when the workload is genuinely opaque, and even then it belongs to the platform, not your request path.

Forces

• Cost vs quality. The cheapest model that clears the bar for a role is the right model for that role — but "clears the bar" is per-role, and a global default optimizes neither end.
• Simplicity vs adaptivity. A static role→model map is trivial to reason about and test; a per-request classifier adapts to inputs but adds a call, latency, and a way to be wrong. Most application traffic does not need the adaptivity.
• Isolation vs sprawl. Model ids, provider slugs, and prices churn constantly. Left inline they leak into dozens of call sites; a breaking change becomes a scavenger hunt.
• Spend ceiling vs availability. A hard budget cap protects the invoice but can hand a user an error mid-task; a soft downgrade keeps the feature alive at lower quality. You want both, on different thresholds.

The pattern

Name your roles — the small, closed set of task classes your product actually makes. Map each role to a model tier in one place. Every call site names a role, never a model. That single function is the only code that knows a model id, which makes model churn a one-line edit and cost a property of the role rather than an accident of where the call was written.

Two things ride alongside the map. First, prompt caching: mark the stable prefix — the system prompt and tool definitions, which are identical across calls in a role — as cacheable, so each call re-reads that prefix at roughly a tenth of the input price instead of paying full freight every turn. Second, fallback routes: on budget pressure, downgrade rather than fail — a soft cap sends traffic to the cheap tier; only a hard cap returns an error, and that error is a graceful in-band message, not a stack trace.

Reference implementation notes

Anthropic

The tiers are the model family: Haiku for the cheap role, Sonnet for the default, Opus for judgment. The router is a plain switch — no framework earns its place here:

// one function knows model ids; every call site names a role
type Role = 'hint' | 'tutor' | 'grader' | 'eval';

function modelFor(role: Role): string {
  switch (role) {
    case 'hint':   return 'claude-haiku-4-5';   // $1 / $5  per MTok
    case 'tutor':  return 'claude-sonnet-5';    // $3 / $15 per MTok
    case 'grader': return 'claude-sonnet-5';
    case 'eval':   return 'claude-opus-4-8';    // $5 / $25 per MTok
  }
}

Prompt caching is the multiplier on top. Mark the system prompt and tool definitions with a cache breakpoint; a cached prefix read bills at roughly 10% of the input rate, so a long stable system prompt stops being a per-turn tax. Anthropic's default cache lives ~5 minutes with a 1-hour option, and batch submission halves the rate again for anything that need not be live. The one trap that survives a model swap: tokenizers drift. Sonnet 5 renders the same text as roughly 30% more tokens than the prior generation, so a cost estimate baselined on an old tokenizer silently under-reads after an upgrade — re-measure token counts, do not assume them.

AWS (Bedrock)

When the workload genuinely is opaque — you cannot name the role at the call site — this is exactly where a difficulty router earns its place, and AWS ships it as a managed service so you do not build one. Amazon Bedrock Intelligent Prompt Routing (generally available) predicts, per request, which model in a family will meet the quality bar at the lowest cost and dispatches accordingly; AWS reports up to ~30% cost reduction on suitable workloads. Two constraints shape when to use it: it routes within a single model family, and it is optimized for English prompts. Combine it with Bedrock prompt caching for the same stable-prefix saving. Use the managed router for opaque traffic; keep the static role map for the calls whose class you already know — they are not mutually exclusive.

Cloudflare (AI Gateway)

AI Gateway is the operational half: it sits in front of the provider and gives routing, caching, and spend control without touching call-site code. Dynamic routing composes a flow (visual or JSON) that evaluates conditions, enforces quotas, and picks a model with fallbacks — a primary node whose fallback points at a second model, on a different provider if you like, that fires when the primary errors or times out. Response caching serves identical requests from Cloudflare's edge, cutting latency and repeat-call cost, with per-request TTL control. Spend rules live in the dashboard: a hard monthly cap returns 429 so the app can degrade in-band; a soft rule can dynamically route down to the cheap tier. This is the gateway layer doing budget-pressure fallback for you, above whatever static map your code already carries. (Note: the older Universal Endpoint is deprecated in favor of dynamic routing — build new flows there.)

Trade-offs

• The map needs maintenance. A static role→model map is only as good as its last review. New roles get bolted onto an existing tier out of laziness; model upgrades change the price/quality frontier. Revisit the map on every model release, not never.
• Roles can multiply. The pattern is clean with a handful of roles and muddy with thirty. If you find yourself minting a role per call site, the abstraction has inverted — collapse them back to task classes.
• Caching adds coupling. Prompt caching only pays when the prefix is genuinely stable; interleave volatile data into the cached region and you thrash the cache and pay the write premium for nothing. It constrains how you order a prompt.
• Soft downgrade hides regressions. A fallback to the cheap tier keeps the feature alive but silently lowers quality; without a per-call cost-and-model log you will not notice you have been serving the budget tier for a week.

When not to use this

Skip it when there is genuinely one kind of call. A single-purpose feature — one classification endpoint, one summarizer — has one role, and "routing" over one role is a switch with one arm. Pick the right model once and move on; the map is overhead with nothing to organize.

Skip the static map, specifically, when you cannot name the task class at the call site — open-ended user prompts of unknown difficulty. That is the case a difficulty classifier or a managed router (Bedrock Intelligent Prompt Routing) is built for; forcing a hardcoded role map onto opaque traffic just moves the wrong guess into your code. And do not reach for either flavor of routing to rescue a quality problem that is really a prompt or eval problem — if the top tier does not clear the bar, a router will not, and you are tuning the wrong knob. Fix the prompt and the eval first (see the eval-harness gate), then route.

As-built evidence

aiArch runs this pattern in production. The coach and assessment features route by role through one function — modelFor() in src/lib/llm.ts — with hint on Claude Haiku 4.5, tutor and grader on Claude Sonnet 5, and eval on Claude Opus 4.8. Call sites name a role and never a model, so the tutor role's 4.6→5 upgrade was a one-line change in that file and nothing else. The same file marks two prompt-cache breakpoints — the stable system-plus-tools prefix and a rolling conversation prefix — so multi-turn coaching re-reads its transcript at cache-read prices; src/lib/cost.ts bills cache reads at 0.1× input and logs a per-turn USD figure per route, which is how a silent downgrade would surface.

The whole stack sits behind Cloudflare AI Gateway on the authenticated path. A hard monthly cap returns 429, which the code maps to a BudgetExhaustedError and turns into a graceful in-band coach message rather than a crash; a soft rule downgrades to Haiku via dynamic routing. The routing indirection is deliberate churn-isolation: provider slugs, gateway wiring, and the cache markers all live in the one src/lib/llm.ts boundary, so a breaking upstream change stays a one-file fix. Cost discipline here is not incidental — it is the same Track 0 and Track B lesson the platform teaches, applied to itself.

Changelog

• 2026-07-13 — Initial publication. Pricing and provider-feature state verified against current Anthropic, AWS, and Cloudflare docs.