Pattern · last reviewed 2026-10-19

Semantic caching: the cache that can answer the wrong question

Context & problem

“Enable caching” is one ticket and three different decisions. A prompt-cache marker, a toggle in the gateway dashboard and a vector index in front of the model all get described with the same sentence, cut cost and latency on repeats, and they have almost nothing else in common. On their default keys, the first two cannot show a user an answer meant for someone else. The third can, by design, and nothing in an HTTP 200 will tell you when it did.

The first of the three is covered in depth on cache-aware prompt layout, and the gateway as a product on what Cloudflare AI Gateway is. This page is about the boundary between the second and the third, and how to decide which routes in your own system may sit behind a similarity lookup.

Three caches that share a name

The differences are in what is stored, what a hit returns and what decides that a hit is a hit.

Two vendor details change how you should read the middle column. Cloudflare AI Gateway caching is off until you turn it on, either gateway-wide in the dashboard or per request with a header, and its caching page still lists semantic search for caching as planned work rather than a feature (page updated 30 Sep 2026). So on that gateway, similarity matching is something you build, not something you switch on.

The other detail is that Cloudflare does ship a similarity cache, inside AI Search rather than AI Gateway. It is worth knowing because it shows that “similar” is whatever the vendor says it is. AI Search matches on MinHash and locality-sensitive hashing over overlapping word groups, which is overlap in wording rather than closeness in embedding space. It names its threshold steps from near-identical to loose, defaults to the second step, expires entries after 48 hours unless you change the TTL, and clears the cache when the document chunks the answers were built from change. Same risk class as an embedding cache, a different definition of “close”, and a built-in invalidation rule most hand-built caches lack.

Forces

• The saving and the risk are one dial. Match loosely and the cache fills and fails more often; match tightly and it is safe and nearly empty. Exact match sits at one end, and it is the only point on the dial where a hit is the answer to the same request.
• A similarity score measures closeness of text, not sameness of the answer. A negation, a changed number or a different option letter is a tiny textual difference and a complete change of answer. The right answer also depends on things that are not in the text: who is asking, what they have already done, which passages the corpus held that day, which prompt version ran. If those are not in the key, the cache cannot tell them apart.
• A false hit is silent. It is fluent, on topic, returned fast and logged as a success. The only instrument for it is sampling hits against a fresh, uncached answer.
• The hidden state you add to the key is the hit rate you give up. Put the user, the history and the retrieved passages in the key and the cache becomes safe and almost never hits. This is the same dial again, seen from the other end.
• Repetition has to exist before any of it pays. AWS’s lead example is a help-desk chatbot, where thousands of users ask one question in different words. Workloads where every query is unique get a lookup cost and no hits.

The pattern

Treat the semantic cache as something a route has to earn its way into. Seven steps, in this order.

• Decide per route. A gateway-wide switch or a shared middleware decides for every route at once, including the ones nobody thought about. The unit of admission is a route and its prompt.
• Run the admission test. Fail any of the four and the route gets exact-match caching or none. (1) State: is the answer a function of the question text alone? List everything else the prompt carries: user, memory, history, tool results, retrieved passages, the clock. Each is hidden state. (2) Sensitivity: can a one-word difference change the right answer? If a negation or a different number flips it, closeness in embedding space is the wrong test, and you should try your own pairs against your own embedding model rather than trust a threshold. (3) Audience: would you be content to show this answer to any other user who asks something near it? (4) Consequence: does anything downstream act on the answer as a decision, such as a grade, an authorisation or a moderation verdict?
• Put hidden state into the key as exact filters, before similarity. AWS’s ElastiCache guidance does this with tag filters for user, session, model and product, and says the per-user filter is what stops one user’s cached conversation being returned to another. Add model, prompt version and corpus version to that list, because a changed prompt or a re-indexed corpus makes every stored answer stale at once.
• Embed what the answer depends on. AWS’s multi-turn example is two conversations that both end in “Tell me more”, one about Valkey and one about Python. Embedding the last message alone treats them as the same question. AWS’s best-practices page recommends resolving the conversation first and caching on the current message together with the retrieved context, instead of embedding the entire raw dialogue.
• Start strict and loosen only with evidence. AWS suggests beginning at 0.90 to 0.95 and lowering while monitoring accuracy, with A/B testing. Write the unit next to every number. The same AWS documentation set states its threshold table as similarity, where higher is closer, and its multi-turn code as a maximum cosine distance of 0.12, where lower is closer. A threshold belongs to one embedding model and one metric; re-tune it when either changes.
• Measure false hits, not only hit rate. AWS’s monitoring example counts hits and misses and turns them into estimated savings, which is a number about money. Sample some hits, generate the uncached answer, compare, and alert on the disagreement rate.
• Give every entry an expiry and an invalidation event. A TTL bounds how stale an answer gets. A corpus or prompt change should clear entries immediately, the way AI Search does for its own cache.

The order of operations, as a sketch of shape rather than a library:

# Exact filters first, similarity second, audit on a sample of hits.
scope = { tenant, user, model, promptVersion, corpusVersion }
vec   = embed(contextTheAnswerDependsOn)           # not only the last message
hit   = nearest(vec, where: scope)
if (hit && hit.distance <= MAX_DISTANCE)           # say the unit: distance, not similarity
  if (!sampleForAudit()) return hit.response
  else compare(hit.response, callModel(...))        # count disagreements
reply = callModel(...)
store(vec, scope, reply, ttl)

There is a cheaper alternative worth trying first. Cloudflare’s documentation for its custom cache key says to choose a key that groups “only requests with equivalent responses”, which is the admission test in one line. If you can classify an incoming question into a small closed set of intents and key the cache on the intent, you get approximate matching with an exact-match key and no threshold to tune. That is our suggestion rather than a documented recipe, and the classifier becomes the new place a wrong answer can come from.

Trade-offs

• Every miss now costs more than having no cache. An embedding call and a vector lookup run before the model call. AWS’s illustrative table has a 25% hit ratio taking a 100,000-question-a-day assistant from $945 to $732 a day, a 23% saving, with embedding generation left out of the cost. It also carries the cache cluster at $23 a day. Scale that traffic down by a factor of 100 and the model bill is about $9.45 a day against the same $23 cluster, so the cache costs more than it saves. The figures are AWS’s illustration and the division is ours, but the shape holds: a semantic cache has a fixed cost, and it needs traffic to cover it.
• You are operating a second store of user text. Whatever the cache holds, queries and responses, has its own retention and its own deletion story. We turned off gateway log payloads for exactly this reason: the stored text sat outside both our retention wipe and our erasure routine. A response cache or a vector cache of learner answers would be another such store, outside both routines again.
• The cache is a second place where policy has to hold. Cloudflare’s data-loss-prevention documentation says cache hits skip DLP scanning, which Cloudflare calls safe because only responses that passed or were merely flagged are cached, and that a policy changed after a response was cached does not re-evaluate it; the stale response is served until the TTL ends. Any output filter that runs only on generated text has the same property for your semantic cache.
• Do not rely on it for stampede protection. Cloudflare notes that its cache is volatile: two identical requests sent at the same moment can both miss, because the first has not been stored in time for the second. The AI Search cache says the same of similar requests.
• It couples you to the embedding model. Stored vectors and the threshold are only meaningful together. Changing the embedding model means re-embedding or flushing, and a flush resets your hit rate to zero.

When not to use this

When the answer depends on state the question does not carry. Personalised replies, multi-turn conversation and tool-using agents all fall here. The embedding sees the words; the answer was shaped by a user profile, a history and a tool result. Putting all of it into the key is possible, and then you are back to a cache that almost never hits.

When the output is a decision the system acts on. A grade, an authorisation, a moderation or safety verdict, a routing choice: a near-miss here is not a slightly worse sentence, it is a wrong state change for someone else’s input. Run the model, or use a deterministic check.

When users are not interchangeable. If an answer for one user must not be shown to another, the isolation is a filter you build and test, not something a similarity index gives you. AWS’s per-user example adds that filter explicitly; without it the index is shared.

When a wrong answer costs more than a slow one. AWS places medical, legal and financial applications at its strictest threshold. If you would need the strictest setting to be comfortable, the hit rate that setting leaves is the number to evaluate, not the one in the vendor’s overview.

When freshness is tighter than any TTL you could set. AWS calls real-time and highly dynamic answers often poor candidates. A price, a stock level or a status page is an answer with an expiry in minutes.

When traffic does not repeat. Reviewing a different diff every time, or answering a different document every time, produces no near-duplicates to find. Measure how often a stored query would actually have matched before you build anything.

When the saving you want is the prefix. A long shared system prompt or tool list is what prompt caching is for, and it carries no correctness risk at all. Reach for the semantic cache only for the cost that remains after that.

When the same gateway carries calls that must stay fresh. Cloudflare’s AI Search documentation says not to enable AI Gateway caching on the gateway that serves an AI Search instance, because cached embeddings can quietly degrade retrieval. A gateway-wide cache setting reaches calls you were not thinking about when you turned it on.

As-built: five model routes, no response cache of ours on any of them

This is what the code on this platform does and does not do, which is narrower than what we believe about our infrastructure. Where the repository cannot show something, that is said.

What the code sends. Every LLM chat call goes through headers() in src/lib/llm.ts (lines 213 to 254). It always sends the provider bearer token. On the AI Gateway path it adds cf-aig-authorization when a gateway token is set, cf-aig-collect-log-payload: false and, when known, cf-aig-metadata carrying a surface route label and the learner id. It never sends cf-aig-cache-key, cf-aig-cache-ttl or cf-aig-skip-cache; a search of src/, test/ and scripts/ on 2026-10-19 finds none of them. There is also no response cache of our own: no code path embeds a prompt and looks up a stored answer. The only Vectorize upsert calls, in src/lib/rag.ts and src/lib/guideSearch.ts, index lesson and guide passages so that retrieval can find evidence for an answer. They do not store answers.

What we do cache. On the model path, provider prompt caching, and only that. applyCacheControl in src/lib/llm.ts (lines 355 to 367, called at 376 to 378) puts two breakpoints on anthropic/-routed models. By Anthropic’s own statement that cannot change a reply, which is why it is the one cache we run without an admission test.

What the repository cannot show. The gateway’s “Cache Responses” setting is dashboard state, and nothing in this repo reads it. Cloudflare documents caching as off by default; when it is enabled for a gateway, its default applies to every request that supports caching and per-request headers override it. Because our code sends none of the three headers, it neither opts in nor opts out. If someone switched the dashboard setting on, our calls would be eligible. Closing that is one line in headers() and it is not in the code today, so our not caching rests on a default we do not assert. The same limit applies to spend rules: the code absorbs a gateway 429, but whether any rule is configured is also dashboard state.

Why exact-match gateway caching would not leak, and would not help. Cloudflare’s default key includes the provider authorisation header, and we send one app-wide provider key for every learner (llm.ts:218), so that component separates nothing. Separation rests entirely on the request body. On the coach and the graders the body carries per-learner state, so two learners’ requests are rarely byte-identical, and when they are, nothing in the body tells them apart. An exact cache is therefore safe there, and it would save close to nothing. That last part is inference from what the bodies contain, not a measurement.

The more interesting question is what a similarity cache would do on each route, so here is the four-question admission test run against the real request bodies.

The last row is the instructive one. Guide-coach is exactly the shape AWS lists as a good fit: a shared knowledge base asked the same thing in different words by many users, here anonymous visitors. We have not built a cache for it, and the reason is its ceiling. The route is capped at 215 live calls a day, sized so that the worst case is about $1 a day (guideCoach.ts:36-50, which also notes that real spend is at or below that). A cache cannot save more than the route can spend. Against a $1 ceiling, an embedding call, an index to keep in step with the guide corpus and a false-hit audit are all cost with nothing to offset them.

A sixth caller, the editorial drafter in src/lib/drafter.ts (lines 147 to 166), is out of scope: its input is a vendor page that has just changed and its output waits for a human decision.

What we do not have is a measurement. We have built no semantic cache and request no gateway cache, so we have no hit rate and no false-hit rate for any route, and the reasoning above is a design position, not an experiment. The case for revisiting it is volume on the guide route rising enough that the ceiling stops being the answer.

Changelog

• 2026-10-19 — Initial publication. Cloudflare AI Gateway caching, its DLP interaction, AI Search similarity cache and the AI Search gateway guidance read from developers.cloudflare.com; the semantic caching overview, best practices and multi-turn pages from the Amazon ElastiCache user guide and the savings table from the AWS Database Blog’s vector-search announcement; the exact-match and output-identity statements from Anthropic’s prompt-caching documentation. All retrieved 2026-10-19.