Retrieval-augmented generation (RAG)

Grounding an LLM's answer in documents fetched at query time: the ingest, retrieve, augment, generate pipeline, its add-ons (query rewriting, HyDE, contextual retrieval), its CAG alternative and how to evaluate it.

Contents

Retrieval-augmented generation (RAG) answers a question by first retrieving relevant passages from an external corpus and then letting the LLM generate from those passages placed in its prompt. The model’s weights are the parametric memory; the index is a non-parametric memory that can be updated, swapped or permission-filtered without retraining (Lewis et al. 2020). It is the default way to ground answers in private, large or fast-changing data and to reduce hallucination by giving the model something to cite (Notebook).

Origin

The term comes from Lewis et al. (NeurIPS 2020): a DPR bi-encoder retriever over about 21M 100-word Wikipedia passages, a BART-large generator, and only the query encoder and generator fine-tuned. Two variants: RAG-Sequence conditions the whole answer on one set of passages, RAG-Token may draw on a different passage per token. Their key demonstration: replacing the 2016 Wikipedia index with a 2018 one updated the model’s answers about world leaders without any retraining (source). Modern RAG keeps the idea but usually drops the joint training: a frozen, off-the-shelf retriever and a frozen chat LLM glued by a prompt (Synthesis).

The pipeline end to end

flowchart LR
  subgraph "Ingestion (offline)"
    A["Raw docs (PDF, HTML, tickets)"] --> B["Parse + clean + metadata"]
    B --> C["Chunk"]
    C --> D["Contextualize chunk (optional)"]
    D --> E["Embed (dense)"]
    D --> F["Index terms (BM25)"]
  end
  subgraph "Query time"
    Q["User question"] --> R["Rewrite / HyDE / decompose"]
    R --> S["Hybrid search: dense + BM25"]
    E --> S
    F --> S
    S --> T["Fuse (RRF) top ~100-150"]
    T --> U["Rerank (cross-encoder) keep top ~5-20"]
    U --> V["Prompt: instructions + chunks + question"]
    V --> W["LLM answer with citations"]
  end

Synthesis: the diagram assembles the stages described by Anthropic and the Notebook; no single source shows exactly this graph.

  1. Ingestion. Parse documents (document processing), split them into chunks, embed each chunk with an embedding model into a vector database and, for hybrid search, also build a keyword index. The owner’s experiment 13 is the minimal version: split, embed, FAISS, top-k, grounded answer.
  2. Retrieval. Encode the query and find nearest chunks with an ANN index, ideally fused with BM25 (hybrid search), then rerank a wide candidate set down to a few.
  3. Augmentation. Put the chunks in the prompt with an instruction to answer only from them and to cite. Order matters: models use information at the start and end of the context better than the middle (Lost in the Middle), so put the best chunks at the edges and don’t pad with marginal ones.
  4. Generation. The LLM answers; citations let users and evaluators check grounding.

Query-side techniques

TechniqueFixesMechanismCost
Query rewriting (Rewrite-Retrieve-Read)Vague, conversational or multi-part queriesAn LLM rewrites the query before search; a small rewriter can be RL-trained on reader feedback (Ma et al.)One extra LLM call
HyDEShort queries embed poorly next to long documentsAn LLM writes a hypothetical answer; embed that and search (Gao et al.)One generation per query; fake details can steer retrieval
Decomposition / multi-queryQuestions needing several factsSplit into sub-queries, retrieve for each, merge (Notebook)N retrievals
Contextual retrievalChunks that lost their document contextPrepend an LLM-written 50-100 token blurb to each chunk before embedding and BM25 (Anthropic)One-time, about $1.02 per M doc tokens (2024 pricing, with prompt caching)

Anthropic’s measurements (failure rate = 1 - recall@20) show how the stages stack: 5.7% baseline, 3.7% with contextual embeddings, 2.9% adding contextual BM25, 1.9% adding a reranker, a 67% reduction overall (source).

RAG vs CAG vs long context

Cache-augmented generation (CAG) skips retrieval: preload the whole (small) knowledge base into the prompt once, precompute its KV cache, and answer every query from that cached prefix (Chan et al.). With hosted APIs this is long context plus prompt caching (Notebook). Anthropic’s rule of thumb: under about 200k tokens (about 500 pages), just put it all in the prompt (source).

RAGCAG
Corpus sizeUnboundedMust fit the context window
FreshnessUpdate the index any timeRebuild the cache on change
Main failureMissed or wrong retrievalOverflow, staleness, long-context degradation
Sees whole documentNo, only chunksYes
InfraEmbeddings + vector DB (+ BM25, reranker)Model + caching

Evidence: on HotPotQA and SQuAD subsets up to 85k tokens with Llama 3.1 8B, CAG beat sparse and dense RAG in five of six settings, but sparse RAG (top-5) won on the largest HotPotQA set (85k tokens; BERTScore 0.7535 vs 0.7407). The authors note the CAG advantage narrows as the corpus grows, citing prior findings on long-context degradation, but do not analyse that specific result (Chan et al.). The common production shape mixes them: cache the stable core (policies, glossary), retrieve the volatile long tail (tickets, catalog) (Notebook, experiment 14).

Variants for harder questions

  • GraphRAG for multi-hop and whole-corpus (“main themes?”) questions that no single chunk answers.
  • Agentic RAG: retrieval becomes a tool the model calls iteratively; includes corrective RAG (grade, then fall back) and Self-RAG (model decides when to retrieve and critiques support). Agents increasingly retrieve “just in time” with tools such as grep instead of only pre-computed embeddings, at the cost of speed (Anthropic).

Evaluating RAG

Evaluate retrieval and generation separately, since “retrieval is the ceiling”: if the right chunk never arrives, the generator cannot recover (Notebook).

  • Retrieval: recall@k / failure rate on a labelled set of (question, relevant chunk) pairs, as Anthropic did (source).
  • Generation (RAGAS): faithfulness = supported claims / total claims in the answer; answer relevance = mean cosine similarity between the question and questions an LLM regenerates from the answer; context relevance = share of context sentences actually needed (Es et al.). The library adds context precision, context recall (needs references), noise sensitivity and an HHEM classifier option for cheaper faithfulness checks, though that option sits in the legacy metrics API that Ragas has said it will remove in 1.0 (Ragas docs). In the paper, faithfulness agreed with humans 95% of the time; context relevance only 70% (Es et al.).

These are LLM-as-judge metrics, so they inherit judge bias and cost; see LLM evaluation.

Failure modes

SymptomLikely causeFix
Correct doc exists, never retrievedChunk lost context; exact term (ID, SKU, error code) missed by dense searchContextual retrieval, hybrid search
Right chunk retrieved but ranked 30thBi-encoder imprecisionReranking, pass 20 rather than 5 chunks (Anthropic)
Answer ignores a supplied chunkLost in the middle, too many chunksFewer, better-ordered chunks (Liu et al.)
Confident answer from irrelevant contextNo relevance gateCorrective RAG / grading (agentic RAG)
“Summarize everything” failsGlobal question, top-k can’t cover itGraphRAG
Stale answersIndex not refreshedIncremental re-ingest

Synthesis: two more production failure modes worth designing for are access control (filter by the user’s permissions at retrieval time, not after generation) and prompt injection through retrieved documents, which should be treated as data, not instructions.

When to use / when not

  • Use RAG when the corpus is large, changes often, needs per-user filtering, or answers need citations (Notebook).
  • Prefer CAG / long context when the corpus is small (roughly under 200k tokens) and stable (Anthropic).
  • Prefer fine-tuning for changing behaviour, format or style rather than adding facts (Synthesis, consistent with the Notebook’s note that fine-tuning facts is slow and hard to update).
  • Start simple and add hybrid, rerank or graph machinery only when evals show it helps (Notebook).

Frameworks: LlamaIndex and LangChain provide the plumbing; text-to-SQL is the structured-data counterpart.

Sources

Suggest changes →