Tokenization

How LLMs split text into subword tokens (BPE, byte-level BPE, Unigram, WordPiece, SentencePiece), why token counts differ across vendors, models and languages, and how to count them correctly.

Contents

Tokenization is the step that converts text into the integer ids a transformer actually reads, and converts generated ids back into text. Modern LLMs use subword tokenizers: frequent words become a single token, rare words split into several known pieces, and nothing is ever “out of vocabulary” (Hugging Face; Sennrich et al.). The token is the unit of everything operational: the context window is measured in tokens, APIs bill per token, and latency grows with tokens generated (notebook).

Why subwords

GranularityVocabularySequence lengthProblem
WordHuge (every inflection)ShortUnseen words map to <unk>; giant embedding matrix
CharacterTinyVery longEach unit carries little meaning; quality suffers
SubwordFixed, typically ~50k-200kModerateCompromise used by LLMs

(Hugging Face; vocabulary range from the notebook)

Subword vocabularies came into neural NLP to solve open-vocabulary translation: names, compounds and loanwords can be built from smaller units (Sennrich et al.).

The algorithms

flowchart LR
    A["Raw text"] --> B["Normalise / pre-tokenise (split on spaces, digits, punctuation)"]
    B --> C["Apply learned model: BPE merges or Unigram segmentation"]
    C --> D["Token strings"]
    D --> E["Map to ids; add special tokens (BOS, role markers)"]
    E --> F["Model"]

Synthesis: generic pipeline; SentencePiece skips the whitespace pre-tokenisation step and works on the raw stream.

AlgorithmHow the vocabulary is builtSegmentation at inferenceUsed by (per sources)
BPEStart from characters; repeatedly merge the most frequent adjacent pair; vocab = base + mergesDeterministic: replay merge rulesLlama, Gemma, Qwen2 (HF)
Byte-level BPESame, but base vocabulary is the 256 byte valuesAny UTF-8 input encodable, no <unk>GPT-2 (50,257 entries) (HF); OpenAI’s tiktoken (tiktoken)
WordPieceMerge the pair maximising freq(ab) / (freq(a)·freq(b))Synthesis: greedy longest-match against the vocabularyBERT family (HF)
Unigram LMStart large; prune tokens whose removal least increases lossProbabilistic; can sample alternative segmentationsT5, BigBird, Pegasus (HF)
SentencePiece (library)Runs BPE or Unigram on raw text; space becomes ▁Fully reversible; language-independentLanguages without spaces (SentencePiece; HF)

BPE as used by OpenAI is reversible and lossless, handles arbitrary text, and averages about 4 bytes per token (tiktoken).

Vocabulary size is a trade-off. A larger vocabulary compresses text into fewer tokens (cheaper, more fits in context) at the cost of a bigger embedding table. Llama 3 moved to a 128K vocabulary and gets up to 15% fewer tokens than Llama 2 (Llama 3). OpenAI’s encodings went from cl100k_base to o200k_base (tiktoken).

Rules of thumb (and where they break)

  • English: about 4 characters per token; roughly 750 words per 1,000 tokens (notebook). Google gives about 4 characters and 60-80 words per 100 tokens for Gemini (Gemini).
  • Claude Opus 4.7 and later (including the Fable/Mythos families and Mythos Preview) use a newer tokenizer that produces roughly 30% more tokens for the same text, with the exact increase depending on content (Anthropic). On it, 1M tokens is about 555k words, versus about 750k words on earlier Claude models (Claude models overview). Rules of thumb are tokenizer-specific.
  • Other languages cost more. Translations of the same text can differ by up to 15x in token count, even with multilingual tokenizers; that means higher cost, higher latency and less content per window (Petrov et al.).
  • Code, numbers, rare words and unusual whitespace tokenize into more pieces (notebook).
  • Media are tokens too. Gemini charges 258 tokens per small image (or per 768×768 tile) and 32 tokens per second of audio (Gemini).

Counting tokens across vendors (as of 2026-11)

VendorHow to count before sendingNotes
OpenAILocal: tiktoken.encoding_for_model(...), then len(enc.encode(text))Open-source library; encoding per model (tiktoken)
AnthropicAPI: POST /v1/messages/count_tokens with the same body as a Messages requestFree, rate-limited, an estimate; counts tools, images, PDFs, thinking; rejects server tools (except the advisor tool), MCP connector and url/file media; does not apply prompt caching (Anthropic)
Google GeminiAPI: countTokens (input only); response usage reports input, output, thought, cached and tool-use tokens(Gemini)
Open-weight modelsLoad the model’s own tokenizer (e.g. Hugging Face)Synthesis: each family ships its own vocabulary

Two traps:

  1. A raw-text count is not the request count. Requests are transformed into a model-ready format before reaching the model (Synthesis: role markers, tool schemas) and outputs are parsed afterwards, so billed tokens do not match visible content one-to-one (Messages API). Anthropic may count system-added tokens that it does not bill (Anthropic).
  2. Counts don’t transfer between tokenizers. Recount against the exact model you’ll deploy, especially on a migration (Anthropic). Synthesis: never estimate Claude or Gemini usage with tiktoken except as a rough order of magnitude.

Failure modes caused by tokenization

  • Arithmetic and digits. GPT-3.5/GPT-4 tokenizers chunk numbers into up-to-3-digit tokens from the left, misaligning with place value; forcing right-to-left chunking with thousands separators markedly improves addition (Singh & Strouse).
  • Character-level tasks (spelling, counting letters, reversing strings). Synthesis: the model sees ids for multi-character chunks, not letters, so such tasks need tools or explicit decomposition. See LLM limitations.
  • Language inequity in cost and context, as above (Petrov et al.).
  • Budget surprises after a model upgrade from a tokenizer change (Anthropic).
  • Stop sequences and token boundaries. Synthesis: a stop string can span token boundaries; vendors match on decoded text, but behaviour around whitespace can surprise. See sampling parameters.

Practical guidance

Synthesis:

  • Budget with the target model’s counter and keep a margin (10-20%) for formatting overhead.
  • For multilingual products, measure tokens per language on real data before pricing; consider languages in model selection.
  • Prefer formats that tokenize compactly for large payloads (e.g. trimmed JSON, CSV) when feeding data; it saves cost and context. See cost and latency optimisation.
  • When numbers matter, format them consistently (separators) or hand arithmetic to a tool (Singh & Strouse).
  • Embeddings models have their own tokenizers and input limits; chunk sizes for chunking should be measured in that model’s tokens.

Sources

Suggest changes →