For engineers running AI in production

LLM cost optimization: token budgeting, caching, and routing

Why input tokens dominate

The first instinct is to compare per-token prices and pick the cheapest model. That's the wrong frame for an agent. The cost shape of a multi-turn loop is what matters: on each turn the model gets the entire conversation so far — system prompt, tool definitions, retrieved context, and every prior message — re-sent as input. As the run grows, the input side of every call grows with it. Output is usually a small fraction by comparison.

So the bill is dominated by the same large prefix being re-read turn after turn. For reference, current Claude rates per 1M tokens (input / output):

Notice the output rate is the larger number per model — but in a multi-turn agent you re-send far more input than you generate output, so the input column is where the money goes. Optimize the thing that scales with every turn.

Sonnet 5 is priced at $2 / $10 (introductory pricing made permanent 10 Aug 2026) but ships a new tokenizer that emits roughly 30% more tokens for the same text than Sonnet 4.6, so re-measure token counts on the model you actually run rather than carrying an older per-request estimate forward.

Measure first: counting tokens before you optimize

You can't optimize what you haven't measured. Before you cache, route, or compact anything, size the prompt and context you're actually sending — guessing leads you to optimize the wrong thing. The right tool here is a token counter, and Anthropic ships one as a first-class endpoint: POST /v1/messages/count_tokens (in the SDKs, messages.countTokens / messages.count_tokens). It returns an estimate of the input_tokens a request would bill, and it understands the full request shape — system prompts, tool definitions, images, PDFs, and extended thinking all count.

Two things make it the right measurement tool. It's free — you're charged nothing for counting, subject only to its own separate rate limits (tier 1 is 100 requests per minute). And it counts against the tokenizer of whichever model id you pass, which matters more than it used to: Opus 4.7+ and Fable 5 use a newer tokenizer that emits roughly 30–35% more tokens for the same text. Count with the actual target model id; don't reuse a count taken against a pre-4.7 model or you'll under-budget by a third.

The levers

1. Prompt caching

Most of what you re-send each turn is static: the system prompt, the tool definitions, and any fixed context. Cache that prefix once and repeated turns read it from cache instead of re-billing it at the full input rate. Cache reads bill at roughly 0.1x the input rate (0.05x on Claude Opus 5.5; 0.025x on Claude Fable 5.1 and Mythos 5.1); the trade is that the initial cache write costs about 1.25x the input rate. With a stable prefix re-read across many turns, that one-time premium pays for itself quickly. This is the single biggest lever for a long-running agent because it attacks the exact cost — the re-sent prefix — that dominates the bill.

2. Model routing

Not every sub-task needs your strongest model. Route work down the ladder — Opus to Sonnet to Haiku — and send cheap or simple steps (classification, extraction, routing decisions, short formatting) to a smaller, cheaper model. Reserve the expensive model for the steps that actually need its reasoning. The price spread above makes this material: Haiku input is one-fifth of Opus input.

3. Turn and tool-call caps with done-conditions

An agent that doesn't know when it's finished will keep looping, and every loop re-sends the growing context. Set an explicit maximum on turns and tool calls, and define a clear done-condition so the agent stops the moment the goal is met. This is the guardrail that prevents the worst-case bill, not the average one.

4. Context management and compaction

Dead context is pure overhead: you pay to re-send it every turn for no benefit. Prune resolved tool outputs, summarize or compact older turns, and keep only what the next step needs. Smaller carried context means a smaller input side on every subsequent call.

5. Batch where latency allows

For work that isn't latency-sensitive — overnight processing, bulk classification, offline evals — batch the requests. You trade immediacy for a lower effective rate, and a large share of production LLM work doesn't actually need a real-time response.

6. Instrument per-route cost

You can't cut what you can't see. Attribute token usage and cost to each route, endpoint, or agent so the hotspot is obvious. Per-route cost turns "the bill went up" into "this one flow is 80% of spend" — which is what tells you where to apply the levers above.

The levers at a glance

Prompt caching: the multiplier math

Caching is the biggest lever, so it's worth knowing the exact multipliers rather than the "roughly 0.1x" shorthand (0.05x on Claude Opus 5.5; 0.025x on Claude Fable 5.1 and Mythos 5.1). Every cache mode prices as a multiple of the model's base input rate. The write is a one-time premium; the read is the discount you collect on every subsequent turn that hits the same prefix.

The architect's read: pay the 1.25x write once, then bill the static prefix at a tenth of input on every turn thereafter (a twentieth on Opus 5.5, a fortieth on Fable 5.1 and Mythos 5.1) — that's the whole win. Reach for the 1-hour TTL only when the gap between reuses exceeds the 5-minute window; otherwise you're paying 2.0x to write for a benefit the cheaper 5-minute cache already gives you. And because the batch discount stacks, latency-tolerant bulk work over a cached prefix compounds both savings: 0.1x reads (0.05x on Opus 5.5, 0.025x on Fable 5.1 and Mythos 5.1) at 50% off. For how these multipliers play out over a full agent run, see what a Claude agent actually costs to run.

As-built: the rate table above is literally our own router

The Haiku/Sonnet/Opus rate table earlier on this page isn't a generic reference — it's the exact pricing table in our own src/lib/cost.ts ($1/$5, $2/$10, $5/$25 per MTok), and the routing decision it feeds is modelFor() in src/lib/llm.ts: a plain switch routing our hint role to Haiku, tutor/grader to Sonnet, and safeguard — the coach's input-injection judge — to the open-weight gpt-oss-safeguard-20b rather than an Anthropic tier. A fifth role, eval (Claude Opus 5), is declared and reserved for capstone evaluation but has no live call site yet. We didn't build a classifier or a cost-aware trained router — our request classes are known at the call site, so a static rule is the whole router.

On the AI-Gateway side, the code path is live in production behind an authenticated Cloudflare AI Gateway: any gateway-path 429 maps to a BudgetExhaustedError and degrades to an in-band message rather than crashing the stream. Which spend-limit rules are configured on our gateway is dashboard state, not repo state, so we claim no specific rule. We isolated all of this behind one file (src/lib/llm.ts) so a provider or gateway change is a one-file fix, not a rewrite. See the model-router pattern for the full write-up.

The two pieces of spend we did not intend were both gate failures, not routing failures. Every lever above assumes a call you meant to make. Ours were calls nobody chose:

• An ambient key turned a test run into a purchase. Our live eval suites were guarded with "skip unless an API key is present" — which is the natural way to write it, and wrong, because the key is permanently exported in the owner's shell. A plain npm test therefore made real API calls for roughly 200 seconds with no prompt and no confirmation. The documentation said to prefix the command to disable it; a documented prefix is not a guard, it is a thing you forget once. The suites now gate on an explicit opt-in signal (test/liveGate.ts) — the npm lifecycle event or an environment flag set for that run — so key presence alone can never spend. Wall-clock duration was the tell that found it: a test suite that suddenly takes minutes instead of seconds is billing you.
• A dry-run caught a 2.6× overspend before it happened. Our offline media scripts require an explicit --confirm-spend flag on top of key presence; without it they resolve the full payload, print it, and stop. On the very first run, that dry-run showed 680 characters of narration where 261 were intended — heuristic markdown stripping had leaked the script file's metadata block into the text to be synthesised. Per-character billing means we would have paid 2.6× for audio that read a header aloud, and it is the class of bug you cannot detect until after you have paid for it. The generator now requires explicit start/end markers and refuses rather than guessing.

The generalisable rule is unglamorous: put the confirmation on the spend, not on the environment. Presence of a credential is not intent, and a warning in a README cannot fail a run. In code, that is one flag check before the call and a printed estimate when the flag is absent.

Frequently asked questions

How do you reduce LLM cost?

Attack the input side, because a multi-turn agent re-sends its growing context every turn and input tokens — not output, not the headline rate — dominate the bill. The three highest-leverage moves: cache the large static prefix so repeated turns bill at roughly 0.1x the input rate (0.05x on Claude Opus 5.5; 0.025x on Claude Fable 5.1 and Mythos 5.1), route cheap sub-tasks down the model ladder (Opus to Sonnet to Haiku), and cap turns and tool calls with explicit done-conditions so a runaway loop can't quietly run up the meter. Bound the loop first, cache second, route third; the per-token rate is the last thing to worry about.

How do you track LLM cost per route or per user?

Attribute token usage and cost to each route, endpoint, or agent, so “the bill went up” becomes “this one flow is 80% of spend” and you know where to apply the levers. Measure before you optimize: the free POST /v1/messages/count_tokens endpoint sizes a request against the tokenizer of the model id you pass. On this site the model calls run behind an authenticated Cloudflare AI Gateway, and a gateway-path 429 maps to a BudgetExhaustedError that degrades in-band instead of crashing the stream.

Which LLM is the most cost-effective?

The wrong frame — cost-effectiveness is per role, not per model. Not every sub-task needs your strongest model: send classification, extraction, routing decisions and short formatting to a smaller, cheaper tier and reserve the expensive one for steps that need its reasoning. The spread is material (Haiku input is one-fifth of Opus input), and when the request classes are known at the call site a static rule is the whole router; see the model-router pattern.

Does prompt caching reduce latency?

Yes, as well as cost. Anthropic's prompt-caching docs describe caching prefixes as a way to “cut costs and latency”, and name conversational agents with long instructions or uploaded documents as the case where both drop. The 5-minute and 1-hour caches behave the same with respect to latency, so the TTL choice is purely the cost trade above: 1.25x versus 2.0x to write, 0.1x to read on every hit either way (0.05x on Opus 5.5, 0.025x on Fable 5.1 and Mythos 5.1).