Retrieval-augmented generation (RAG)
Grounding an LLM's answer in documents fetched at query time: the ingest, retrieve, augment, generate pipeline, its add-ons (query rewriting, HyDE, contextual retrieval), its CAG alternative and how to evaluate it.
Contents
Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from an external corpus and then letting the LLM generate from those passages placed in its prompt. The model’s weights are the parametric memory; the index is a non-parametric memory that can be updated, swapped or permission-filtered without retraining (Lewis et al. 2020). It is the default way to ground answers in private, large or fast-changing data and to reduce hallucination by giving the model something to cite (Notebook).
Origin
The term comes from Lewis et al. (NeurIPS 2020): a DPR bi-encoder retriever over about 21M 100-word Wikipedia passages, a BART-large generator, and only the query encoder and generator fine-tuned. Two variants: RAG-Sequence conditions the whole answer on one set of passages, RAG-Token may draw on a different passage per token. Their key demonstration: replacing the 2016 Wikipedia index with a 2018 one updated the model’s answers about world leaders without any retraining (source). Modern RAG keeps the idea but usually drops the joint training: a frozen, off-the-shelf retriever and a frozen chat LLM glued by a prompt (Synthesis).
The pipeline end to end
flowchart LR
subgraph "Ingestion (offline)"
A["Raw docs (PDF, HTML, tickets)"] --> B["Parse + clean + metadata"]
B --> C["Chunk"]
C --> D["Contextualize chunk (optional)"]
D --> E["Embed (dense)"]
D --> F["Index terms (BM25)"]
end
subgraph "Query time"
Q["User question"] --> R["Rewrite / HyDE / decompose"]
R --> S["Hybrid search: dense + BM25"]
E --> S
F --> S
S --> T["Fuse (RRF) top ~100-150"]
T --> U["Rerank (cross-encoder) keep top ~5-20"]
U --> V["Prompt: instructions + chunks + question"]
V --> W["LLM answer with citations"]
endSynthesis: the diagram assembles the stages described by Anthropic and the Notebook; no single source shows exactly this graph.
- Ingestion. Parse documents (document processing), split them into chunks, embed each chunk with an embedding model into a vector database and, for hybrid search, also build a keyword index. The owner’s experiment 13 is the minimal version: split, embed, FAISS, top-k, grounded answer.
- Retrieval. Encode the query and find nearest chunks with an ANN index, ideally fused with BM25 (hybrid search), then rerank a wide candidate set down to a few.
- Augmentation. Put the chunks in the prompt with an instruction to answer only from them and to cite. Order matters: models use information at the start and end of the context better than the middle (Lost in the Middle), so put the best chunks at the edges and don’t pad with marginal ones.
- Generation. The LLM answers; citations let users and evaluators check grounding.
Query-side techniques
| Technique | Fixes | Mechanism | Cost |
|---|---|---|---|
| Query rewriting (Rewrite-Retrieve-Read) | Vague, conversational or multi-part queries | An LLM rewrites the query before search; a small rewriter can be RL-trained on reader feedback (Ma et al.) | One extra LLM call |
| HyDE | Short queries embed poorly next to long documents | An LLM writes a hypothetical answer; embed that and search (Gao et al.) | One generation per query; fake details can steer retrieval |
| Decomposition / multi-query | Questions needing several facts | Split into sub-queries, retrieve for each, merge (Notebook) | N retrievals |
| Contextual retrieval | Chunks that lost their document context | Prepend an LLM-written 50-100 token blurb to each chunk before embedding and BM25 (Anthropic) | One-time, about $1.02 per M doc tokens (2024 pricing, with prompt caching) |
Anthropic’s measurements (failure rate = 1 - recall@20) show how the stages stack: 5.7% baseline, 3.7% with contextual embeddings, 2.9% adding contextual BM25, 1.9% adding a reranker, a 67% reduction overall (source).
RAG vs CAG vs long context
Cache-augmented generation (CAG) skips retrieval: preload the whole (small) knowledge base into the prompt once, precompute its KV cache, and answer every query from that cached prefix (Chan et al.). With hosted APIs this is long context plus prompt caching (Notebook). Anthropic’s rule of thumb: under about 200k tokens (about 500 pages), just put it all in the prompt (source).
| RAG | CAG | |
|---|---|---|
| Corpus size | Unbounded | Must fit the context window |
| Freshness | Update the index any time | Rebuild the cache on change |
| Main failure | Missed or wrong retrieval | Overflow, staleness, long-context degradation |
| Sees whole document | No, only chunks | Yes |
| Infra | Embeddings + vector DB (+ BM25, reranker) | Model + caching |
Evidence: on HotPotQA and SQuAD subsets up to 85k tokens with Llama 3.1 8B, CAG beat sparse and dense RAG in five of six settings, but sparse RAG (top-5) won on the largest HotPotQA set (85k tokens; BERTScore 0.7535 vs 0.7407). The authors note the CAG advantage narrows as the corpus grows, citing prior findings on long-context degradation, but do not analyse that specific result (Chan et al.). The common production shape mixes them: cache the stable core (policies, glossary), retrieve the volatile long tail (tickets, catalog) (Notebook, experiment 14).
Variants for harder questions
- GraphRAG for multi-hop and whole-corpus (“main themes?”) questions that no single chunk answers.
- Agentic RAG: retrieval becomes a tool the model calls iteratively; includes corrective RAG (grade, then fall back) and Self-RAG (model decides when to retrieve and critiques support). Agents increasingly retrieve “just in time” with tools such as grep instead of only pre-computed embeddings, at the cost of speed (Anthropic).
Evaluating RAG
Evaluate retrieval and generation separately, since “retrieval is the ceiling”: if the right chunk never arrives, the generator cannot recover (Notebook).
- Retrieval: recall@k / failure rate on a labelled set of (question, relevant chunk) pairs, as Anthropic did (source).
- Generation (RAGAS): faithfulness = supported claims / total claims in the answer; answer relevance = mean cosine similarity between the question and questions an LLM regenerates from the answer; context relevance = share of context sentences actually needed (Es et al.). The library adds context precision, context recall (needs references), noise sensitivity and an HHEM classifier option for cheaper faithfulness checks, though that option sits in the legacy metrics API that Ragas has said it will remove in 1.0 (Ragas docs). In the paper, faithfulness agreed with humans 95% of the time; context relevance only 70% (Es et al.).
These are LLM-as-judge metrics, so they inherit judge bias and cost; see LLM evaluation.
Failure modes
| Symptom | Likely cause | Fix |
|---|---|---|
| Correct doc exists, never retrieved | Chunk lost context; exact term (ID, SKU, error code) missed by dense search | Contextual retrieval, hybrid search |
| Right chunk retrieved but ranked 30th | Bi-encoder imprecision | Reranking, pass 20 rather than 5 chunks (Anthropic) |
| Answer ignores a supplied chunk | Lost in the middle, too many chunks | Fewer, better-ordered chunks (Liu et al.) |
| Confident answer from irrelevant context | No relevance gate | Corrective RAG / grading (agentic RAG) |
| “Summarize everything” fails | Global question, top-k can’t cover it | GraphRAG |
| Stale answers | Index not refreshed | Incremental re-ingest |
Synthesis: two more production failure modes worth designing for are access control (filter by the user’s permissions at retrieval time, not after generation) and prompt injection through retrieved documents, which should be treated as data, not instructions.
When to use / when not
- Use RAG when the corpus is large, changes often, needs per-user filtering, or answers need citations (Notebook).
- Prefer CAG / long context when the corpus is small (roughly under 200k tokens) and stable (Anthropic).
- Prefer fine-tuning for changing behaviour, format or style rather than adding facts (Synthesis, consistent with the Notebook’s note that fine-tuning facts is slow and hard to update).
- Start simple and add hybrid, rerank or graph machinery only when evals show it helps (Notebook).
Frameworks: LlamaIndex and LangChain provide the plumbing; text-to-SQL is the structured-data counterpart.
Sources
- Lewis et al. 2020: Retrieval-Augmented Generation
- Anthropic: Introducing Contextual Retrieval
- Notebook: RAG vs CAG vs GraphRAG
- Notebook experiment 13: RAG
- Notebook experiment 14: RAG + CAG
- Gao et al. 2022: HyDE
- Ma et al. 2023: Query Rewriting
- Liu et al. 2023: Lost in the Middle
- Chan et al. 2024: Cache-Augmented Generation
- Es et al. 2023: RAGAS
- Ragas docs: Available metrics
- Anthropic: Effective context engineering