Source: Anthropic: Introducing Contextual Retrieval
Anthropic engineering post showing that LLM-written chunk context plus BM25 plus reranking cuts retrieval failures by 67%.
Contents
- What: vendor engineering blog post
- Author: Anthropic (Daniel Ford)
- Date: 2024-09-19
- Original: Introducing Contextual Retrieval
Summary
Argues that standard chunking strips the context a chunk needs to be found, and proposes prepending a short, chunk-specific context (written by a small LLM that sees the whole document) to every chunk before embedding it and before BM25 indexing. Reports retrieval-failure reductions for each added stage and gives practical advice on top-K, rerankers and embedding models.
Key claims
- For a knowledge base under about 200k tokens (about 500 pages), skip RAG and put it all in the prompt; prompt caching makes this cheaper (up to 90%) and faster (over 2x).
- Contextual Embeddings and Contextual BM25: each chunk gets a 50 to 100 token context blurb generated by Claude 3 Haiku from the full document.
- One-time cost estimated at $1.02 per million document tokens, assuming 800-token chunks, 8k-token documents and prompt caching (2024 pricing).
- Failure rate (1 minus recall@20): baseline 5.7%; contextual embeddings 3.7% (-35%); plus contextual BM25 2.9% (-49%); plus reranking 1.9% (-67%).
- Reranking setup: retrieve the top 150 chunks, rerank with Cohere, keep the top 20; passing 20 chunks beat 10 or 5.
- Gemini and Voyage embeddings performed best; contextual retrieval helped every embedding model tested.
- Adding generic document summaries to chunks gave very limited gains.
Feeds
Retrieval-augmented generation · Chunking · Hybrid search · Reranking