Pattern · last reviewed 2026-08-14

Cache-aware prompt layout: the cheapest edit becomes the most expensive

Context & problem

A multi-turn agent re-sends its own history. Turn eight carries the system prompt, the tool definitions and turns one through seven, so input tokens accumulate and dominate the bill — the arithmetic is worked through on what a Claude agent actually costs to run, and the multiplier table lives on LLM cost optimization. Neither page is about the thing that decides whether you collect the discount, which is where in the prompt the boundary sits.

The mental model most teams start with is "cache the system prompt". It is close enough to be dangerous. What actually gets cached is a prefix: Anthropic's prompt-caching documentation states that caching "references the entire prompt — tools, system, and messages (in that order) up to and including the block designated with cache_control". So the unit is not a field. It is everything from the start of the request to your marker, in a fixed render order, and a single differing byte anywhere in that span is a miss.

That produces a decision most codebases make by accident: for every piece of context you assemble — a user id, a timestamp, a retrieved document, a per-request instruction — you are choosing whether it sits before the boundary and freezes, or after it and stays free. Nobody writes that decision down, because in an uncached prompt it does not exist.

Forces

• Stability versus readability. The order that caches best is volatility order. The order a human maintains best is topic order. They are rarely the same order, and only one of them is enforced by anything.
• The discount is asymmetric in time. You pay the write premium immediately and collect the read discount later, inside a window. Anything that lengthens the gap between requests — a slow tool, a thinking user, a nightly job — attacks the collection side only.
• The minimum is a cliff, not a slope. Below the model's minimum cacheable length nothing caches at all, and the request succeeds anyway.
• Cheap models are harder to cache. The minimum does not track price, so the model you route high-volume work to may be the one your prefix cannot clear.
• Observability is vendor-shaped. The only evidence a cache hit happened is a usage field, and the three API surfaces you are likely to call report it in three different shapes — one of which has no field for the write at all.

The pattern

Four decisions, in this order.

• Sort the prompt by change frequency, and write the reason down. Tools and system instructions change on deploy. Retrieved context changes per session. User input, timestamps and per-turn state change every request. Emit them in that order. The comment that says why a block is where it is matters more than the block — it is the only thing standing between your layout and the next well-meaning refactor.
• Put the breakpoint on the last block that is identical across the requests you want to share a cache — not on the last block, and not on "the system prompt" by reflex. Anthropic's own guidance is blunt about the failure this prevents: "The lookback does not find stable content behind your breakpoint and cache it. It finds entries that prior requests already wrote, and writes happen only at breakpoints." A marker sitting one block past the boundary — on the block carrying the timestamp — writes a fresh entry every request and reads none, forever.
• Check the prefix clears the model's minimum, per model. The minimums are non-monotonic in price: 512 tokens for Opus 5, 1,024 for Sonnet 5, and 4,096 for Haiku 4.5. Below them, "Any requests to cache fewer than this number of tokens will be processed without caching, and no error is returned." Anthropic's suggested remedy is the counter-intuitive one and it is correct: if you are close, pad the cached region up to the threshold rather than trimming it down.
• Instrument the hit before you believe in it. The cache leaves no trace anywhere except the usage object. If your cost dashboard cannot show cache-read tokens per route, you have bought an optimisation on faith and you will keep paying for it after it stops working.

A second breakpoint is usually worth it in a conversation and it is not the expensive part: "Adding more cache_control breakpoints doesn't increase your costs", since you are billed on what is actually cached and read. The ceiling is four per request. The cost of the pattern is the ordering commitment, not the markers.

Reference implementation notes

The layout discipline is identical everywhere. What differs is where the marker goes, what the failure looks like, and what the platform will tell you afterwards.

Managed: first-party API, Bedrock, and the gateways in between

On the Claude API you can place cache_control on individual blocks, or set it once at the top level and let the system apply the breakpoint to the last cacheable block. Automatic placement is the right default when your prompt genuinely ends with stable content, and precisely the wrong one when it does not — the documentation names the trap directly, since automatic caching "places the breakpoint on the last cacheable block, which in this structure is the one that changes every request". If your final block is the incoming user message, place the breakpoint yourself.

AWS's Bedrock prompt-caching documentation calls the same marker a cache checkpoint and matches the model on the important points — four checkpoints per request, a 5-minute default TTL, and the same silent failure below the minimum: "If you try to add a cache checkpoint before meeting the minimum number of tokens, your inference will still succeed, but your prefix will not be cached." Two Bedrock specifics are worth knowing before you port a working layout to it. Its per-model minimums are published separately from Anthropic's and its own table lists Claude Haiku 4.5 at 4,096 tokens per checkpoint. And prompt caching there is "only supported for on-demand inference endpoints. It is not supported with the batch inference API" — so the two cheapest levers you have, caching and batch, do not compose on that platform, which is not obvious from either feature's own page.

If you reach the model through an OpenAI-compatible gateway rather than the native API, the layout is unchanged but the accounting is not. See the failure mode below; this is the seam that costs people money quietly.

Rolling your own

There is nothing to build. That is the point worth making, because it is why this pattern gets skipped in design review: it produces no component, no service and no diagram box, so it never gets an owner. What it needs instead is a rule someone is accountable for, and the cheapest durable form is a single prompt-assembly function that takes the stable and volatile parts as separate arguments and emits them in order. Not for elegance — so that the ordering constraint has one place to live and one place to be reviewed, instead of being an emergent property of five call sites.

Trade-offs

• Your prompt is now ordered by economics, and the file does not say so. A block sits at position three because it is stable, not because it reads well there. That reasoning is invisible to everyone downstream, and the refactor that breaks it — moving a paragraph, inlining a variable, adding a request id to the system message for tracing — is exactly the kind of change that gets waved through as cosmetic. This is the pattern's real cost and it is a people cost, not a token cost.
• Editing the stable half now has a price, and the price falls on the edit you most want to make. Every change to the cached prefix invalidates it for every live session, and the next request from each of them pays the write premium again. That is a small, real, recurring argument against improving your system prompt. Notice it and decide deliberately; teams that do not notice it simply stop editing.
• Invalidation cascades, and some of the triggers are not text. The render order is tools → system → messages, and changes at one level invalidate that level and every level after it. Modifying any tool definition — a name, a description, one parameter — invalidates the entire cache. So does toggling web search or citations, because those rewrite the system prompt. Adding or removing an image, or changing tool_choice, invalidates the message cache. Your prompt text can be byte-identical and the cache still cold.
• Parallel fan-out inverts the saving. This is the one that surprises people who reason about it from first principles, because a set of workers sharing one system prompt looks like the ideal caching case. It is not, on the first pass: "a cache entry only becomes available after the first response begins. If you need cache hits for parallel requests, wait for the first response before sending subsequent requests." Fire five workers simultaneously and you pay five writes at 1.25× — 6.25 units of prefix where a serialised version pays 1.25 + 4 × 0.1 = 1.65. Roughly 3.8× worse, for the thing you did to make it faster. Sequence the first call, then fan out.
• The window is measured from the request that writes, not from when you finished reading the answer. The 5-minute clock "is measured from the start of the request that writes or reads the cache entry, not from the end of its response", and generation time counts against it — a four-minute streamed response leaves about one minute. For a long-running agent turn, the TTL you think you have is not the TTL you have.

The failure mode: three usage shapes, and a saving you cannot see

Caching does not fail as an error. It fails as a cost regression, and the only instrument that can detect one is the usage object — which is reported differently by every surface you might call.

# Anthropic native — ADDITIVE. The three fields sum to the real total.
{ "input_tokens": 1000, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4000 }

# Bedrock Converse — additive, different names, and it names the write.
{ "inputTokens": 1000, "cacheWriteInputTokens": 0, "cacheReadInputTokens": 4000 }

# OpenAI-compatible gateway — INCLUSIVE. cached_tokens is a SUBSET of prompt_tokens,
# and there is no field for the write at all.
{ "prompt_tokens": 5000, "prompt_tokens_details": { "cached_tokens": 4000 } }

Two consequences follow, and neither is stated on any vendor's page, because each page describes only its own shape.

First: read the direction wrong and your cost dashboard is wrong in a way that survives review. Anthropic documents its fields as summing to the total, so billing input_tokens at 1× and adding the cache read is correct. Do the same arithmetic against an OpenAI-compatible payload and you bill the 4,000 cached tokens twice — once inside prompt_tokens at full rate and once again as a cache read. Both readings produce a plausible number, which is why the error is durable. We hit this seam directly: src/lib/llm.ts subtracts cached_tokens from prompt_tokens on the way in, specifically so the rest of the codebase can assume the additive semantics, and the correction carries a test that pins the direction.

Second, and worse: on the OpenAI-compatible shape the cache write is invisible. There is no cache_creation field to map, so write tokens arrive folded into prompt_tokens and get billed internally at 1× when the vendor charged 1.25×. Our own accounting has a correct 1.25× branch in src/lib/cost.ts that, on the live route, never executes — because nothing upstream can populate the field that triggers it. The error is small and it fails in the safe direction, which is exactly why it can sit there for months: internal cost figures read slightly low, and no alarm exists for "cheaper than expected".

Third, from our own tree: the resilience path degrades you to full price at HTTP 200. When a request 400s with a body mentioning cache, src/lib/llm.ts retries the call without the cache markers. That is the right behaviour — a coach turn that answers at full price beats a coach turn that errors — and it means a provider-side shape change turns the optimisation off permanently, with a single console.warn as the entire signal. Any degradation path that silently removes a cost optimisation needs a counter behind it, or you will find out from the invoice.

Add the two mechanisms the vendor does document but almost nobody has read, and the list of ways to lose the discount without seeing anything is five long: the prefix under the model's minimum ("no error is returned"), the breakpoint one block past the boundary, the 20-block lookback window — "The system checks at most 20 positions per breakpoint", so a growing conversation can push your breakpoint out of range of the last write and simply stop hitting — the invisible invalidators above, and a strip-and-retry of your own making.

When not to use this

When the prompt is still being written. The write premium only amortises across reads of an unchanged prefix. A system prompt in active iteration invalidates on every deploy, so you pay the surcharge, collect little of the discount, and — the part that actually costs you — acquire a quiet financial argument against editing the text you most need to edit. Ship the caching after the prompt settles, not before. The ordering discipline is free and worth adopting on day one; the markers can wait.

When your traffic is spaced wider than the TTL — and note that the break-even is not where intuition puts it. Do the arithmetic in units of the prefix at base input rate. Caching N requests that share a prefix inside one window costs 1.25 + 0.1 × (N − 1); not caching costs N. Those cross at N ≈ 1.28, so the second hit inside the window has already paid for the write. The risk was never the arithmetic — it is the window. If your calls are spaced further apart than the TTL, every single one is a write, and caching is a permanent 25% surcharge on your prefix rather than a saving. That is the shape of the low-traffic internal tool most of this audience actually operates: a dozen uses a day, spread out. Reaching for the 1-hour TTL is the natural move and it is usually the wrong one, because it doubles the write premium to 2× to buy a window that is still shorter than your gap. Measure the real inter-request interval first; if you cannot, you are not ready to enable this.

When you are routing high-volume work to the cheapest model. This interaction is the one to check before anything else on this page, because the two levers are almost always recommended together and their minimums do not agree. Claude Haiku 4.5 requires 4,096 tokens before anything caches; Claude Sonnet 5 requires 1,024, and Claude Opus 5 only 512. So the minimum runs opposite to price, and a 2,000-token prefix that caches happily on Sonnet 5 caches never on Haiku 4.5 — with no error, no warning and a bill that is merely disappointing rather than alarming. A model router that pushes cheap sub-tasks down a tier can therefore move exactly your highest-volume traffic onto the one model where your prefix is uncacheable. Either pad the cached region to clear the higher floor, or accept that the cheap tier runs uncached and stop counting a saving you are not collecting.

When you are treating the cache as a tenancy boundary. Isolation is a deployment property here, not a code property, which is why it never surfaces in a code review. Anthropic states that "Caches are also isolated per workspace within an organization on the Claude API, Claude Platform on AWS, and Microsoft Foundry; Bedrock and Google Cloud use organization-level isolation only." Read that as an architecture constraint rather than a footnote: a design that leaned on per-workspace separation to keep two tenants' cached context apart loses that separation on a platform move, and not one line of your code changes. If the separation is load-bearing, enforce it with something you control — separate prefixes per tenant, or no shared prefix at all — and treat the platform's isolation as defence in depth rather than as the boundary.

And not as a substitute for sending less. Caching makes re-sending the same tokens cheap; it does not make it free, and it does nothing at all for context you should have dropped. Prune the dead context first — that is agent memory's job — then cache what genuinely must ride along on every turn. A 90% discount on tokens you did not need is still a bill.

As-built: how this platform lays out its coach prompt

We run this one, and the interesting part is that the discipline shows up in three different files as three different-looking decisions.

The coach's system prompt is a module constant in src/lib/coach.ts — frozen at build time, identical for every learner, changed only by a deploy. Three pieces of per-turn state that a naive implementation would have appended to it are deliberately routed elsewhere, and the field comments in that file say why in one line each: the per-objective mastery snapshot, the per-learner memory line and the server-composed item context are each marked volatile — rides in the user message, never the system prompt. That is the whole pattern, expressed as three comments rather than as a component.

src/lib/llm.ts places two breakpoints: one on the system message, which caches the tools-plus-system prefix, and a rolling one on the last plain-text conversation message, so each turn re-reads the accumulated transcript at cache-read prices instead of full input. The markers are only attached on anthropic/-routed slugs, so the coach's open-weight injection judge never requests caching it would not get.

Three honest limitations, because the page is worth less without them. We cannot see our own cache writes — we call through an OpenAI-compatible gateway, so the write field described in the failure-mode section does not exist on our path, and the 1.25× branch in src/lib/cost.ts is unreachable in production. Our own cheapest route is the one most at risk — the hint role runs on Claude Haiku 4.5, whose 4,096-token minimum is the highest of the three models we route to, and it is the shortest prompt we send. That is the router interaction above, in our own tree, and we are naming it rather than claiming we cleared it. And the strip-and-retry path is ours, with a warn line and no counter; it is described above as a general failure mode because we built the general failure mode.

What we would fix first, if this were the week's work, is not the layout. It is the instrument: a per-route cache-read ratio, so that all three of those limitations become numbers instead of paragraphs. The layout is already right, and being right about a layout you cannot measure is a claim, not a result.

Changelog

• 2026-08-14 — Initial publication. Prefix semantics, breakpoint placement, the four-breakpoint ceiling, the 20-block lookback, per-model minimums (Opus 5 512 / Sonnet 5 1,024 / Haiku 4.5 4,096), the invalidation cascade, the 5-minute TTL clock, the parallel-request note and the workspace-versus-organization isolation statement all quoted from Anthropic's prompt-caching documentation, retrieved 2026-08-14. Checkpoint minimums, the silent sub-minimum failure and the batch-inference exclusion quoted from AWS's Bedrock prompt-caching documentation, retrieved the same day.