Source: Anthropic: Introducing Contextual Retrieval

Anthropic engineering post showing that LLM-written chunk context plus BM25 plus reranking cuts retrieval failures by 67%.

Contents

Summary

Argues that standard chunking strips the context a chunk needs to be found, and proposes prepending a short, chunk-specific context (written by a small LLM that sees the whole document) to every chunk before embedding it and before BM25 indexing. Reports retrieval-failure reductions for each added stage and gives practical advice on top-K, rerankers and embedding models.

Key claims

  • For a knowledge base under about 200k tokens (about 500 pages), skip RAG and put it all in the prompt; prompt caching makes this cheaper (up to 90%) and faster (over 2x).
  • Contextual Embeddings and Contextual BM25: each chunk gets a 50 to 100 token context blurb generated by Claude 3 Haiku from the full document.
  • One-time cost estimated at $1.02 per million document tokens, assuming 800-token chunks, 8k-token documents and prompt caching (2024 pricing).
  • Failure rate (1 minus recall@20): baseline 5.7%; contextual embeddings 3.7% (-35%); plus contextual BM25 2.9% (-49%); plus reranking 1.9% (-67%).
  • Reranking setup: retrieve the top 150 chunks, rerank with Cohere, keep the top 20; passing 20 chunks beat 10 or 5.
  • Gemini and Voyage embeddings performed best; contextual retrieval helped every embedding model tested.
  • Adding generic document summaries to chunks gave very limited gains.

Feeds

Retrieval-augmented generation · Chunking · Hybrid search · Reranking

Suggest changes →