Pattern · last reviewed 2026-08-10

RAG with grounded citations: making retrieval answers provable

Context & problem

A plain RAG architecture retrieves some chunks, stuffs them into the prompt, and asks the model to answer. It usually works, which is the trap. The failure mode is silent: the model blends retrieved context with its own parametric memory, produces a fluent answer, and gives you no way to tell which sentences came from your documents and which it invented. When retrieval returns nothing relevant, the model answers anyway — from training data, confidently, and often wrongly. You ship it, an auditor or an angry customer finds the one fabricated clause, and now you cannot even reconstruct where the answer came from.

Grounding is the axis that separates RAG from a search box with a language model bolted on. The senior instinct — treat retrieval as a cache in front of the model — gets you most of the way, but caching has no analogue for provenance: the question "can this answer cite its source?" This pattern is the discipline that makes the answer yes.

Forces

• Verifiability vs fluency. The most fluent answer freely mixes sources and invention. A grounded answer is sometimes terser and refuses more — you trade polish for provability.
• Recall vs precision at retrieval. Retrieve too little and you refuse questions you could have answered; retrieve too much and you bury the supporting chunk in noise the model has to re-read every turn.
• Chunk size vs citation granularity. Big chunks retrieve more context but cite imprecisely ("somewhere in this page"); small chunks cite a sentence but fragment meaning across boundaries.
• Cost vs coverage. Reranking, larger top-k, and groundedness evals all cost tokens or latency. The budget is real; the failure of skipping them is invisible until it is expensive.
• Answering vs refusing. A system that never refuses is a system that hallucinates on empty retrieval. Refusal is a feature, and it has to be designed, not hoped for.

The pattern

Grounded RAG is a pipeline with two non-negotiable checkpoints that ordinary RAG omits: a grounding contract at generation (the model must attach a source pointer to each claim, drawn only from retrieved chunks) and a refusal gate before it (if retrieval returns nothing above a relevance floor, do not call the model to answer). Around those sit the pieces you already know — chunk, embed, index, retrieve — but each is tuned to serve citation, not just recall.

Chunk to citation size, not just retrieval size. The chunk is the smallest thing you can cite. If a chunk spans three unrelated paragraphs, a citation to it proves nothing. Split on semantic boundaries (headings, sentences, list items) so a returned chunk is a coherent, quotable unit. Respect the embedding model's token ceiling — chunks longer than the model encodes are silently truncated, and the tail never gets embedded (see the Cloudflare note below for a concrete ceiling we hit in production).

Carry metadata and filter on it. Every chunk keeps its source id, section, timestamp, and any tenant/permission key. Metadata filters run before or alongside vector search so you never retrieve — or cite — a document the user is not allowed to see or that is out of date. In multi-tenant systems this is also a security boundary, not only a relevance tool.

Make grounding a contract, not a request. "Please cite your sources" in a prompt is a suggestion the model can ignore. The stronger form passes each retrieved chunk as a first-class citable block so the platform parses citations into structured pointers with the exact quoted span — the model cannot emit a citation to text that was not retrieved. Where that primitive is not available, the fallback is: instruct the model to answer only from the provided chunks, tag each sentence with a chunk id, and post-validate that every tagged id was actually retrieved.

Refuse on empty retrieval. The gate is the cheapest reliability win in the pattern. If nothing clears a relevance floor, return "I don't have a source for that" instead of calling the model to improvise. This single branch removes the largest class of RAG hallucinations — the ones that happen when there was nothing to ground on in the first place.

Reference implementation notes

The pattern is platform-independent; the grounding primitive differs by stack. Use the first-class one where it exists.

Anthropic (Claude API)

Claude's Citations feature is the grounding contract as a platform primitive. Pass each RAG chunk as a document or search_result content block with citations: {"enabled": true}; the model returns claims interleaved with citation objects carrying cited_text and a location (char_location, page_location, or content_block_location) pointing back into the exact block. Because the API parses citations into a standard format, a citation is guaranteed to point at real provided text — the model cannot fabricate a source. cited_text does not count toward output tokens. All active models support it except Claude Haiku 3. The search_result block is purpose-built for RAG: pass your retrieved results as first-class citable content rather than concatenating them into one document.

# one retrieved chunk as a citable search result
{
  "type": "search_result",
  "source": "https://docs.internal/billing#refunds",
  "title": "Refund policy",
  "content": [{ "type": "text", "text": "Refunds are issued within 14 days." }],
  "citations": { "enabled": true }
}

One constraint to design around: citations cannot be combined with structured outputs — enabling both on a user-provided block returns a 400. If you need a strict JSON schema and citations, separate the two calls.

AWS (Bedrock Knowledge Bases)

Bedrock Knowledge Bases give you managed grounded RAG. RetrieveAndGenerate retrieves, generates, and returns citations tying spans of the answer to retrievedReferences (source location + content) — and it only cites sources relevant to the query. Prefer Retrieve (retrieval only, you own generation) when you need the grounding contract enforced in your own prompt or want to run the refusal gate yourself; use RetrieveAndGenerate when the managed loop is enough. Note the older flat citation member is deprecated in favor of generatedResponse + retrievedReferences. Chunking strategy (fixed / hierarchical / semantic, managed or custom-Lambda) is the lever that sets your citation granularity here.

Cloudflare (AI Search + Vectorize)

AI Search (formerly AutoRAG) runs hybrid retrieval (semantic + BM25, fused and reranked) over an R2 source with continuous re-index; Vectorize is the vector store when you want to drive retrieval directly, with metadata indexes and namespaces for the per-tenant filtering above. The hard-won as-built lesson: the @cf/baai/bge-base-en-v1.5 embedding model truncates input at 512 tokens (~2000 characters of English, fewer for code). Chunk before embedding — a chunk longer than that loses its tail entirely, so anything past the cutoff is never retrievable and can never be cited. Create Vectorize metadata indexes before the first upsert; vectors written earlier are not covered by a filter added later.

Trade-offs

• More refusals. A correctly gated system says "no source" on questions a chattier one would have guessed at. For low-stakes UX that reads as worse; for anything auditable it is the whole point. Tune the floor; do not remove the gate.
• Citation is not correctness. A grounded answer proves a claim traces to a retrieved chunk — not that the chunk is right, current, or relevant. Garbage in the index cites cleanly to garbage. Provenance shifts the quality problem to your corpus, it does not erase it.
• Chunking becomes a first-order design decision. You now tune chunk boundaries for citation granularity as well as recall, and re-chunking means re-embedding the whole corpus. Get the boundary discipline right early.
• Token and latency overhead. Passing chunks as citable blocks, reranking, and running groundedness evals all cost. It is a real budget line — spent to buy verifiability you would otherwise discover you lacked at the worst moment.
• Eval is non-optional. Grounding you never measure is grounding you cannot trust. Groundedness (is each claim supported by a retrieved chunk?) needs its own eval, typically LLM-as-judge over held-out queries — which is another system to build and maintain.

When not to use this

Skip grounded RAG — or RAG entirely — when the situation removes the need for provenance:

• The corpus fits in the context window and is stable. Long-context stuffing of a small, slow-changing set of documents is cheaper and simpler than a retrieval pipeline, and with citable blocks you still get grounding. Retrieval earns its complexity only when the corpus is too large or too fresh to stuff.
• A wrong answer costs nothing. Brainstorming, draft generation, casual internal search where humans obviously verify — the ceremony of citation and refusal is overhead with no payoff. Ground where being wrong is expensive, not everywhere.
• The knowledge is behavioral, not factual. If what you need is a tone, a format, or a skill rather than retrievable facts, that is a fine-tuning or prompt problem; retrieval has nothing to ground against.
• You cannot maintain the index. Grounded RAG is only as trustworthy as its corpus is current. Without an owner for freshness and re-indexing, confident citations to stale documents are worse than an honest "I don't know" — the citation lends false authority.

As-built evidence

aiArch runs this pattern on two separate surfaces, and it does not run the same amount of it on both. Chunking to the embedding ceiling, the metadata filter, the relevance floor, and refusal on empty retrieval are all live here; the grounding contract runs only as a prompt instruction — the fallback's per-sentence tagging and post-validation do not run; the per-claim citation pointer does not run at all. Each claim below names the file and the symbol behind it, so you can check it rather than take it.

The refusal gate is a branch, not a prompt — on the public surface. The ungated "Ask the guides" widget (POST /api/guide-coach, src/routes/guideCoach.ts) retrieves through searchGuides (src/lib/guideSearch.ts) and hands the hits to assembleGuideAnswer (src/lib/guideAnswer.ts). When retrieval comes back empty, that function returns GUIDE_REFUSAL with grounded: false, and the route returns it without calling the model at all — the refusal is an early return, not an instruction. It fires on empty retrieval, which makes it a real gate only to the extent that retrieval comes back empty on a weak match. Until 2026-07-26 neither guide branch carried a similarity floor, so a thin keyword overlap or a distant nearest neighbour counted as grounding and reached the model; both branches now floor their scores, and they do it with two different constants for a reason worth reading (two branches, two scales, below). Both retrieval entry points also return an empty result for an empty or whitespace query instead of embedding it, because a nearest-neighbour lookup on a meaningless vector returns arbitrary passages that would read as grounding and quietly destroy this exact gate.

Citations are built from what was retrieved, never from what the model wrote. Every widget response carries a citations array of slug, title and URL derived from the retrieved hits themselves, so the citation list a reader clicks can never point at a page retrieval did not return. That guarantee covers the list, not the prose: the model is instructed to name the guides it drew from, and nothing validates those names. Anthropic's primitive binds the model's own emitted citations to provided text; ours binds only the list we build beside the answer. If the live call fails, or if the output scanner blocks the reply, the route falls back to the deterministic answer assembled from those same passages. The degraded path here degrades toward more grounding, not less.

The metadata filter sits on the lesson coach, alongside the older of the two relevance floors. retrieve() (src/lib/rag.ts), reached through the coach's get_lesson_context tool (src/lib/tools.ts), queries Cloudflare Vectorize with a lessonId filter — the filter-before-you-retrieve step above, used here for relevance and structurally identical to the tenant case — then discards anything under a cosine floor of 0.55. Both of those controls hold only while the vector path is serving. Three things drop it back to keyword scoring over the same lesson's passages, which keeps the lesson scoping and loses the floor: a missing binding, a Vectorize fault, and — the common one — every match falling below 0.55 on both the filtered and the cross-module query. So the floor re-routes a weak match more than it discards one. That floor is there for a reason worth stealing: Vectorize returns the top-k nearest vectors regardless of relevance, so a populated index never comes back empty, and without a floor the cross-module fallback beneath it is unreachable code while an off-topic question gets grounded in the current lesson's nearest noise. The 0.55 itself is judgment, not measurement — the code says so in a comment — and it is the kind of number you are supposed to tune against real query logs rather than inherit from a pattern page. Note the honest scale too: this is retrieval over our own course corpus and public guides, not a large multi-tenant estate.

Two branches, two score scales, and the shared constant that would have been the wrong fix. The public widget retrieves two ways — a term-overlap keyword search that always works, and a Vectorize query when the binding and Workers AI are both present — and until 2026-07-26 neither had a floor. On the vector branch that made the refusal gate unreachable in production, not merely loose: Vectorize returns the top-k nearest vectors regardless of relevance, the index is bound, and a daily cron seeds it, so retrieval on that path could not come back empty and the early return above could never fire. The obvious repair was to export the lesson coach's MIN_VECTOR_SCORE and use it in both places. That would have been wrong. The keyword branch's score is a count of distinct query tokens found in the passage — an integer that grows with query length rather than with relevance — and a cosine threshold says nothing about a count. The review that argued it down put the general case better than the specific one: two independent tuning knobs over two different indexes, and exporting the constant means the next person who tunes the lesson floor silently retunes the public widget. That is coupling wearing DRY's clothes, and it is the failure mode of applying DRY to two things that merely look alike.

The measured floor also beat the analogised one. What shipped is a keyword floor of min(2, |queryTokens|) gating the best hit only, and a per-hit cosine floor of 0.55 on the vector branch. Both the min() and the decision to gate only the top hit are measurements, not preferences, and the measurement was taken offline by bundling the real module — no network, no embedding spend. Over twelve off-topic questions, a bare score > 0 grounded six, and four of those on a single incidental token ("how do I write a LangGraph state machine" retrieved a Cloudflare certification guide via the word "state"). Requiring two overlapping tokens drops those four and costs nothing on topic: all fourteen questions drawn from the subjects the refusal message itself advertises score two or better. The min() is what keeps a legitimate one-word question answerable — "evals", "MCP", "CCAR-F" and "certifications" all top out at one overlapping token, so a flat 2 would refuse every single-token query. Gating the top hit rather than every hit is the same kind of finding: an overlap count cannot distinguish a weak supporting passage from a short one, and per-hit pruning dropped the certifications guide's own passage in favour of a hub page. The 0.55 on the vector branch is honestly labelled in the code as seeded from the lesson floor and unmeasured — measuring it costs real embedding spend — with the direction of error named: too high falls through to keyword retrieval, which is the path every test already exercises. Both floors are also written !(score >= floor) rather than score < floor, because the second form silently stops filtering on a NaN score; that shape has its own page, fail empty.

Chunking is tuned to a ceiling we actually hit. chunkForEmbedding (src/lib/rag.ts) splits passages at 1500 characters before embedding, against the 512-token limit of @cf/baai/bge-base-en-v1.5 described above. It prefers a sentence boundary for the split, falls back to a word boundary, and only cuts mid-word if neither lands in the back half of the window — so a chunk stays a quotable unit in the ordinary case rather than always. The margin between 1500 characters and the roughly 2000 the limit allows is deliberate slack, not a measurement.

What we do not run is the per-claim citation contract. Neither surface uses Anthropic's document or search_result citation blocks, and neither post-validates that an individual sentence traces to a retrieved chunk. The widget's citation proves the answer's source set — these guides were retrieved, and nothing else was in front of the model — not that every clause in the answer is supported by one of them. On the lesson coach the empty-retrieval case is weaker still: the tool returns a grounding_unavailable note instructing the coach to tell the learner the topic is not covered here rather than answer from memory, and an instruction is something a model can disregard in a way that an early return cannot. The public widget is where this pattern's refusal gate is structural; the coach is where it is still a prompt.

The per-claim check we do not run on retrieval, we do run on grading — and the shape transfers. The short-answer grader asks a judge to mark each key point hit or missed and to return an evidence field alongside it: a verbatim quote from the learner's answer that supports the hit. For a while the parser only checked that the field was non-empty, which is the same non-check as trusting a model's cited source name. A judge that lifted a phrase out of the rubric, or invented a plausible sentence the learner never wrote, scored the point — grade inflation feeding straight into the mastery average and the spaced-repetition intervals. parseShortAnswerJudge (src/lib/grade.ts) now takes the learner's answer as an input and rejects any claimed hit whose evidence cannot be found in it: a substring test after normalising case, whitespace and curly quotes, deliberately tolerant of transcription drift and deliberately intolerant of paraphrase, because paraphrase is the failure being caught. What happens next is the part worth stealing. A violation is not recorded as a miss — an unverifiable quote is evidence the judge malfunctioned, not evidence the claim is false, and collapsing the two silently penalises the learner. It is returned as a contract violation, which rides the existing re-ask-once retry and then degrades to a learner-fair self-grade prompt, so a judge malfunction never quietly fails or passes anyone.

If you are wiring the per-claim contract this page prescribes and your provider does not hand you one, that is the cheap version: normalise, then ask whether the claimed quote is genuinely a substring of the source it claims to quote. Anthropic's Citations feature is the same guarantee made structural — the citation cannot point anywhere except at text you supplied — and a normalised substring test is the poor relation you can run over any model's output. It buys less than it looks like, though, and the limit is instructive: a quote can be verbatim and still support nothing. Our own adversarial grading fixture is a salad of rubric vocabulary in sentence order, trivially quotable and asserting nothing, and it took a judge-prompt rule rather than the substring check to stop it scoring (the eval-harness gate page covers that failed run). Substring verification proves a quote is real. Whether the real quote supports the claim is still a judgement, and still needs a judge.

Changelog

• 2026-07-13 — Initial publication. Verified against Anthropic Citations docs, Bedrock RetrieveAndGenerate API reference, and Cloudflare AI Search / Vectorize docs.
• 2026-07-26 — Replaced an unspecific "Partial" as-built note with the artifacts behind it, named by file and symbol, and stated plainly which of this page's own prescriptions run here and which do not. Corrected the attribution: the lesson coach's retrieval runs on Cloudflare Vectorize with Workers AI embeddings, not on AI Search. Vendor claims elsewhere on the page are unchanged and still carry their 2026-07-13 check dates.
• 2026-07-26 — Corrected the claim that guide retrieval carries no similarity floor: it was true when written and was made stale the same week by the change that added one to each branch. Added the two-floor account — why a single shared constant would have been the wrong fix, what was measured, and why the vector floor is labelled unmeasured.
• 2026-08-10 — Added the judge-evidence verification we run on the grading path: what a non-empty-only evidence check let through, the normalised substring test that replaced it, why a violation degrades rather than scoring a miss, and the limit of substring verification against a verbatim-but-empty quote.