Chunking

Splitting documents into retrievable units for RAG: fixed, recursive, semantic, LLM and late chunking, the size/overlap trade-off, contextual chunk headers, and how to measure chunk quality.

Contents

Chunking is the ingestion step of RAG that splits each document into smaller pieces, each embedded and indexed as its own retrieval unit. The chunk is simultaneously the unit of matching (its embedding must sit near the queries it answers) and the unit of context (what the LLM actually reads). Those two jobs pull in opposite directions, which is why chunking choice alone changed retrieval recall by up to 9% in Chroma’s benchmark (Chroma), and why chunking “severs context across boundaries” is listed as a core RAG limit (Notebook).

The core trade-off

  • Small chunks (around 100-300 tokens) give a focused embedding and precise matches, but can strip the chunk of what it is about: “revenue grew 3%” without the company or quarter (Anthropic).
  • Large chunks (800+ tokens) keep context but average several topics into one vector and fill the prompt with irrelevant tokens. Chroma’s measurements show this as very low token-level precision (Chroma).
  • Overlap copies text across neighbouring chunks so a sentence cut at a boundary survives somewhere. It raises index size and duplicates tokens in the prompt; Chroma found removing overlap improved IoU (Chroma).

Chunk size is also bounded by the embedding model’s max input length and interacts with tokenization: sizes are usually set in tokens, not characters (Synthesis).

Strategies

StrategyHow it splitsStrengthsWeaknesses
Fixed-size token window (e.g. TokenTextSplitter)Every N tokens, with M overlapTrivial, predictableCuts mid-sentence; OpenAI Assistants default 800/400 had the worst efficiency in Chroma’s test
Recursive character (RecursiveCharacterTextSplitter)Tries separators in order (\n\n, \n, ., ) until pieces fitRespects paragraphs and sentences; best all-rounder in Chroma’s test at 200 tokens, 0 overlapIgnores semantics
Structure-awareHeadings, Markdown/HTML sections, code functions, table rowsChunks match how the author organized meaningNeeds a parser per format (document processing)
Semantic (Kamradt)Embeds sentences, splits where adjacent similarity drops below a percentileTopic-coherent chunksDefault version scored slightly below average; unbounded chunk size unless capped
Cluster semantic (Chroma)Groups sentences to maximize within-chunk similarity under a max sizeHighest precision and IoU at 200 tokensExtra embedding pass
LLM chunkerLLM picks split pointsHighest recall (about 92)Slow and costly; low precision at the sizes tested
Late chunkingEmbed the whole document with a long-context model, then mean-pool token vectors per chunk spanChunk vectors carry document context, no extra trainingNeeds token-level outputs from a long-context embedder

Sources: Chroma rows from Chroma; late chunking from Günther et al.; structure-aware row is Synthesis.

Chroma’s headline numbers (text-embedding-3-large, 5 chunks retrieved): recursive 200/0 recall 88.1 / IoU 6.9; cluster 200 recall 87.3 / IoU 8.0; LLM (GPT-4o) recall 91.9 / IoU 3.9; token 800/400 recall 87.9 / IoU 1.4 (Chroma). Treat these as directional: the dataset is small and synthetic.

Putting context back into the chunk

Three ways to fix the “orphaned chunk” problem without making chunks huge:

flowchart TD
  D["Full document"] --> C1["Split into chunks"]
  C1 --> X["Contextual retrieval: LLM writes 50-100 token blurb per chunk, prepended before embedding + BM25"]
  C1 --> H["Metadata headers: title, section path, date stored with chunk"]
  D --> L["Late chunking: embed whole doc, pool token vectors per chunk span"]
  X --> I["Index"]
  H --> I
  L --> I

Synthesis: the metadata-header branch is a common practice not measured in these sources.

  • Contextual retrieval (Anthropic): Claude 3 Haiku sees the whole document plus the chunk and writes a short situating context; cost about $1.02 per million document tokens with prompt caching (2024 pricing). It cut retrieval failures 35% alone and 49% combined with contextual BM25 (hybrid search). Generic document summaries attached to chunks helped very little (Anthropic).
  • Late chunking gets similar context-awareness inside the embedding model, without an LLM call per chunk (Günther et al.).

Contradiction: this page, citing Günther et al., presents late chunking as giving similar context-awareness to contextual retrieval. The contextual retrieval page, citing Merola & Singh 2025, reports late chunking as model-dependent (from neutral to strongly negative) and slightly less relevant than contextual retrieval, though cheaper to compute.

  • Small-to-big retrieval (Synthesis): index small chunks for matching, but return the parent section to the LLM. Frameworks such as LlamaIndex ship this pattern.

Chunking for other consumers

Chunk size also matters for LLM extraction, not just embedding. In GraphRAG, GPT-4 extracted almost twice as many entity references from 600-token chunks as from 2,400-token chunks; the paper used 600 tokens with 100 overlap (Edge et al.).

Measuring chunk quality

Document-level recall hides waste. Chroma’s token-level metrics are a better lens (Chroma):

  • Recall: fraction of the relevant tokens that were retrieved.
  • Precision: fraction of retrieved tokens that are relevant (what you pay for in the prompt).
  • IoU: Jaccard of retrieved vs relevant tokens; penalizes both misses and padding.

Synthesis: in practice, build a small labelled set from your own documents (questions plus the exact supporting span), sweep chunk size, overlap and splitter, and pick on recall@k at the k you will actually pass to the model, then check end-to-end answer quality with RAG evaluation.

Failure modes

  • Tables split across chunks lose their header row; code split mid-function loses its signature (Synthesis).
  • Pronouns and implicit subjects (“the company”, “it”) make chunks unfindable (Anthropic).
  • Defaults copied from tutorials (800/400) waste prompt tokens without improving recall (Chroma).
  • Changing the chunker requires re-embedding the corpus, so version your ingestion pipeline (Synthesis).

Practical guidance

  • Start with a recursive splitter around 200-400 tokens and little or no overlap; tune separators for your format (Chroma).
  • Prefer structure-aware splitting when documents have real structure (Synthesis).
  • Add contextual retrieval when chunks are not self-explanatory; it pairs with prompt caching to stay cheap (Anthropic).
  • If the corpus fits in the context window (about 200k tokens), skip chunking entirely (Anthropic).

Sources

Suggest changes →