For engineers running AI in production

LLM cost optimization: token budgeting, caching, and routing

Why input tokens dominate

The first instinct is to compare per-token prices and pick the cheapest model. That's the wrong frame for an agent. The cost shape of a multi-turn loop is what matters: on each turn the model gets the entire conversation so far — system prompt, tool definitions, retrieved context, and every prior message — re-sent as input. As the run grows, the input side of every call grows with it. Output is usually a small fraction by comparison.

So the bill is dominated by the same large prefix being re-read turn after turn. For reference, current Claude rates per 1M tokens (input / output):

Notice the output rate is the larger number per model — but in a multi-turn agent you re-send far more input than you generate output, so the input column is where the money goes. Optimize the thing that scales with every turn.

The $3 / $15 above is Sonnet 5's standard rate; an introductory $2 / $10 per 1M runs through 31 Aug 2026, so what you are billed today sits below the table until that window closes. Budget on the standard rate.

Sonnet 5 keeps Sonnet's $3 / $15 standard rate but ships a new tokenizer that emits roughly 30% more tokens for the same text than Sonnet 4.6, so re-measure token counts on the model you actually run rather than carrying an older per-request estimate forward.

Measure first: counting tokens before you optimize

You can't optimize what you haven't measured. Before you cache, route, or compact anything, size the prompt and context you're actually sending — guessing leads you to optimize the wrong thing. The right tool here is a token counter, and Anthropic ships one as a first-class endpoint: POST /v1/messages/count_tokens (in the SDKs, messages.countTokens / messages.count_tokens). It returns an estimate of the input_tokens a request would bill, and it understands the full request shape — system prompts, tool definitions, images, PDFs, and extended thinking all count.

Two things make it the right measurement tool. It's free — you're charged nothing for counting, subject only to its own separate rate limits (tier 1 is 100 requests per minute). And it counts against the tokenizer of whichever model id you pass, which matters more than it used to: Opus 4.7+ and Fable 5 use a newer tokenizer that emits roughly 30–35% more tokens for the same text. Count with the actual target model id; don't reuse a count taken against a pre-4.7 model or you'll under-budget by a third.

The levers

1. Prompt caching

Most of what you re-send each turn is static: the system prompt, the tool definitions, and any fixed context. Cache that prefix once and repeated turns read it from cache instead of re-billing it at the full input rate. Cache reads bill at roughly 0.1x the input rate; the trade is that the initial cache write costs about 1.25x the input rate. With a stable prefix re-read across many turns, that one-time premium pays for itself quickly. This is the single biggest lever for a long-running agent because it attacks the exact cost — the re-sent prefix — that dominates the bill.

2. Model routing

Not every sub-task needs your strongest model. Route work down the ladder — Opus to Sonnet to Haiku — and send cheap or simple steps (classification, extraction, routing decisions, short formatting) to a smaller, cheaper model. Reserve the expensive model for the steps that actually need its reasoning. The price spread above makes this material: Haiku input is one-fifth of Opus input.

3. Turn and tool-call caps with done-conditions

An agent that doesn't know when it's finished will keep looping, and every loop re-sends the growing context. Set an explicit maximum on turns and tool calls, and define a clear done-condition so the agent stops the moment the goal is met. This is the guardrail that prevents the worst-case bill, not the average one.

4. Context management and compaction

Dead context is pure overhead: you pay to re-send it every turn for no benefit. Prune resolved tool outputs, summarize or compact older turns, and keep only what the next step needs. Smaller carried context means a smaller input side on every subsequent call.

5. Batch where latency allows

For work that isn't latency-sensitive — overnight processing, bulk classification, offline evals — batch the requests. You trade immediacy for a lower effective rate, and a large share of production LLM work doesn't actually need a real-time response.

6. Instrument per-route cost

You can't cut what you can't see. Attribute token usage and cost to each route, endpoint, or agent so the hotspot is obvious. Per-route cost turns "the bill went up" into "this one flow is 80% of spend" — which is what tells you where to apply the levers above.

The levers at a glance

Prompt caching: the multiplier math

Caching is the biggest lever, so it's worth knowing the exact multipliers rather than the "roughly 0.1x" shorthand. Every cache mode prices as a multiple of the model's base input rate. The write is a one-time premium; the read is the discount you collect on every subsequent turn that hits the same prefix.

The architect's read: pay the 1.25x write once, then bill the static prefix at a tenth of input on every turn thereafter — that's the whole win. Reach for the 1-hour TTL only when the gap between reuses exceeds the 5-minute window; otherwise you're paying 2.0x to write for a benefit the cheaper 5-minute cache already gives you. And because the batch discount stacks, latency-tolerant bulk work over a cached prefix compounds both savings: 0.1x reads at 50% off. For how these multipliers play out over a full agent run, see what a Claude agent actually costs to run.

As-built: the rate table above is literally our own router

The Haiku/Sonnet/Opus rate table earlier on this page isn't a generic reference — it's the exact pricing table in our own src/lib/cost.ts ($1/$5, $3/$15, $5/$25 per MTok), and the routing decision it feeds is modelFor() in src/lib/llm.ts: a plain switch routing our hint role to Haiku, tutor/grader to Sonnet, and safeguard — the coach's input-injection judge — to the open-weight gpt-oss-safeguard-20b rather than an Anthropic tier. A fifth role, eval (Claude Opus 5), is declared and reserved for capstone evaluation but has no live call site yet. We didn't build a classifier or a cost-aware trained router — our request classes are known at the call site, so a static rule is the whole router.

On the AI-Gateway side, the code path is live in production behind an authenticated Cloudflare AI Gateway: any gateway-path 429 maps to a BudgetExhaustedError and degrades to an in-band message rather than crashing the stream. Which spend-limit rules are configured on our gateway is dashboard state, not repo state, so we claim no specific rule. We isolated all of this behind one file (src/lib/llm.ts) so a provider or gateway change is a one-file fix, not a rewrite. See the model-router pattern for the full write-up.

The two pieces of spend we did not intend were both gate failures, not routing failures. Every lever above assumes a call you meant to make. Ours were calls nobody chose:

• An ambient key turned a test run into a purchase. Our live eval suites were guarded with "skip unless an API key is present" — which is the natural way to write it, and wrong, because the key is permanently exported in the owner's shell. A plain npm test therefore made real API calls for roughly 200 seconds with no prompt and no confirmation. The documentation said to prefix the command to disable it; a documented prefix is not a guard, it is a thing you forget once. The suites now gate on an explicit opt-in signal (test/liveGate.ts) — the npm lifecycle event or an environment flag set for that run — so key presence alone can never spend. Wall-clock duration was the tell that found it: a test suite that suddenly takes minutes instead of seconds is billing you.
• A dry-run caught a 2.6× overspend before it happened. Our offline media scripts require an explicit --confirm-spend flag on top of key presence; without it they resolve the full payload, print it, and stop. On the very first run, that dry-run showed 680 characters of narration where 261 were intended — heuristic markdown stripping had leaked the script file's metadata block into the text to be synthesised. Per-character billing means we would have paid 2.6× for audio that read a header aloud, and it is the class of bug you cannot detect until after you have paid for it. The generator now requires explicit start/end markers and refuses rather than guessing.

The generalisable rule is unglamorous: put the confirmation on the spend, not on the environment. Presence of a credential is not intent, and a warning in a README cannot fail a run. In code, that is one flag check before the call and a printed estimate when the flag is absent.