Pattern · last reviewed 2026-07-13

RAG with grounded citations: making retrieval answers provable

Context & problem

A plain RAG architecture retrieves some chunks, stuffs them into the prompt, and asks the model to answer. It usually works, which is the trap. The failure mode is silent: the model blends retrieved context with its own parametric memory, produces a fluent answer, and gives you no way to tell which sentences came from your documents and which it invented. When retrieval returns nothing relevant, the model answers anyway — from training data, confidently, and often wrongly. You ship it, an auditor or an angry customer finds the one fabricated clause, and now you cannot even reconstruct where the answer came from.

Grounding is the axis that separates RAG from a search box with a language model bolted on. The senior instinct — treat retrieval as a cache in front of the model — gets you most of the way, but caching has no analogue for provenance: the question "can this answer cite its source?" This pattern is the discipline that makes the answer yes.

Forces

• Verifiability vs fluency. The most fluent answer freely mixes sources and invention. A grounded answer is sometimes terser and refuses more — you trade polish for provability.
• Recall vs precision at retrieval. Retrieve too little and you refuse questions you could have answered; retrieve too much and you bury the supporting chunk in noise the model has to re-read every turn.
• Chunk size vs citation granularity. Big chunks retrieve more context but cite imprecisely ("somewhere in this page"); small chunks cite a sentence but fragment meaning across boundaries.
• Cost vs coverage. Reranking, larger top-k, and groundedness evals all cost tokens or latency. The budget is real; the failure of skipping them is invisible until it is expensive.
• Answering vs refusing. A system that never refuses is a system that hallucinates on empty retrieval. Refusal is a feature, and it has to be designed, not hoped for.

The pattern

Grounded RAG is a pipeline with two non-negotiable checkpoints that ordinary RAG omits: a grounding contract at generation (the model must attach a source pointer to each claim, drawn only from retrieved chunks) and a refusal gate before it (if retrieval returns nothing above a relevance floor, do not call the model to answer). Around those sit the pieces you already know — chunk, embed, index, retrieve — but each is tuned to serve citation, not just recall.

Chunk to citation size, not just retrieval size. The chunk is the smallest thing you can cite. If a chunk spans three unrelated paragraphs, a citation to it proves nothing. Split on semantic boundaries (headings, sentences, list items) so a returned chunk is a coherent, quotable unit. Respect the embedding model's token ceiling — chunks longer than the model encodes are silently truncated, and the tail never gets embedded (see the Cloudflare note below for a concrete ceiling we hit in production).

Carry metadata and filter on it. Every chunk keeps its source id, section, timestamp, and any tenant/permission key. Metadata filters run before or alongside vector search so you never retrieve — or cite — a document the user is not allowed to see or that is out of date. In multi-tenant systems this is also a security boundary, not only a relevance tool.

Make grounding a contract, not a request. "Please cite your sources" in a prompt is a suggestion the model can ignore. The stronger form passes each retrieved chunk as a first-class citable block so the platform parses citations into structured pointers with the exact quoted span — the model cannot emit a citation to text that was not retrieved. Where that primitive is not available, the fallback is: instruct the model to answer only from the provided chunks, tag each sentence with a chunk id, and post-validate that every tagged id was actually retrieved.

Refuse on empty retrieval. The gate is the cheapest reliability win in the pattern. If nothing clears a relevance floor, return "I don't have a source for that" instead of calling the model to improvise. This single branch removes the largest class of RAG hallucinations — the ones that happen when there was nothing to ground on in the first place.

Reference implementation notes

The pattern is platform-independent; the grounding primitive differs by stack. Use the first-class one where it exists.

Anthropic (Claude API)

Claude's Citations feature is the grounding contract as a platform primitive. Pass each RAG chunk as a document or search_result content block with citations: {"enabled": true}; the model returns claims interleaved with citation objects carrying cited_text and a location (char_location, page_location, or content_block_location) pointing back into the exact block. Because the API parses citations into a standard format, a citation is guaranteed to point at real provided text — the model cannot fabricate a source. cited_text does not count toward output tokens. All active models support it except Claude Haiku 3. The search_result block is purpose-built for RAG: pass your retrieved results as first-class citable content rather than concatenating them into one document.

# one retrieved chunk as a citable search result
{
  "type": "search_result",
  "source": "https://docs.internal/billing#refunds",
  "title": "Refund policy",
  "content": [{ "type": "text", "text": "Refunds are issued within 14 days." }],
  "citations": { "enabled": true }
}

One constraint to design around: citations cannot be combined with structured outputs — enabling both on a user-provided block returns a 400. If you need a strict JSON schema and citations, separate the two calls.

AWS (Bedrock Knowledge Bases)

Bedrock Knowledge Bases give you managed grounded RAG. RetrieveAndGenerate retrieves, generates, and returns citations tying spans of the answer to retrievedReferences (source location + content) — and it only cites sources relevant to the query. Prefer Retrieve (retrieval only, you own generation) when you need the grounding contract enforced in your own prompt or want to run the refusal gate yourself; use RetrieveAndGenerate when the managed loop is enough. Note the older flat citation member is deprecated in favor of generatedResponse + retrievedReferences. Chunking strategy (fixed / hierarchical / semantic, managed or custom-Lambda) is the lever that sets your citation granularity here.

Cloudflare (AI Search + Vectorize)

AI Search (formerly AutoRAG) runs hybrid retrieval (semantic + BM25, fused and reranked) over an R2 source with continuous re-index; Vectorize is the vector store when you want to drive retrieval directly, with metadata indexes and namespaces for the per-tenant filtering above. The hard-won as-built lesson: the @cf/baai/bge-base-en-v1.5 embedding model truncates input at 512 tokens (~2000 characters of English, fewer for code). Chunk before embedding — a chunk longer than that loses its tail entirely, so anything past the cutoff is never retrievable and can never be cited. Create Vectorize metadata indexes before the first upsert; vectors written earlier are not covered by a filter added later.

Trade-offs

• More refusals. A correctly gated system says "no source" on questions a chattier one would have guessed at. For low-stakes UX that reads as worse; for anything auditable it is the whole point. Tune the floor; do not remove the gate.
• Citation is not correctness. A grounded answer proves a claim traces to a retrieved chunk — not that the chunk is right, current, or relevant. Garbage in the index cites cleanly to garbage. Provenance shifts the quality problem to your corpus, it does not erase it.
• Chunking becomes a first-order design decision. You now tune chunk boundaries for citation granularity as well as recall, and re-chunking means re-embedding the whole corpus. Get the boundary discipline right early.
• Token and latency overhead. Passing chunks as citable blocks, reranking, and running groundedness evals all cost. It is a real budget line — spent to buy verifiability you would otherwise discover you lacked at the worst moment.
• Eval is non-optional. Grounding you never measure is grounding you cannot trust. Groundedness (is each claim supported by a retrieved chunk?) needs its own eval, typically LLM-as-judge over held-out queries — which is another system to build and maintain.

When not to use this

Skip grounded RAG — or RAG entirely — when the situation removes the need for provenance:

• The corpus fits in the context window and is stable. Long-context stuffing of a small, slow-changing set of documents is cheaper and simpler than a retrieval pipeline, and with citable blocks you still get grounding. Retrieval earns its complexity only when the corpus is too large or too fresh to stuff.
• A wrong answer costs nothing. Brainstorming, draft generation, casual internal search where humans obviously verify — the ceremony of citation and refusal is overhead with no payoff. Ground where being wrong is expensive, not everywhere.
• The knowledge is behavioral, not factual. If what you need is a tone, a format, or a skill rather than retrievable facts, that is a fine-tuning or prompt problem; retrieval has nothing to ground against.
• You cannot maintain the index. Grounded RAG is only as trustworthy as its corpus is current. Without an owner for freshness and re-indexing, confident citations to stale documents are worse than an honest "I don't know" — the citation lends false authority.

As-built evidence

Partial. aiArch's coach agent runs a modest instance of this pattern in production: before answering, it retrieves the relevant module context from Cloudflare AI Search / Vectorize and grounds its reply in that retrieved material rather than free-associating over the whole curriculum. It is not the full citation-pointer contract described above — the coach grounds on retrieved context but does not yet surface per-claim source spans to the learner — but the retrieve-then-ground spine, and the 512-token chunking discipline the bge-base-en-v1.5 ceiling forces, are the same ones this page prescribes.

Changelog

• 2026-07-13 — Initial publication. Verified against Anthropic Citations docs, Bedrock RetrieveAndGenerate API reference, and Cloudflare AI Search / Vectorize docs.