Context window

The maximum number of tokens a model can attend to in one request (input plus output), what counts against it, vendor limits as of 2026-11, overflow behaviour, and why the usable window is smaller than the advertised one.

Contents

The context window is the total number of tokens a model can reference in a single request, including the response it generates. It is the model’s working memory, distinct from what it learned in training (Anthropic). Everything the model uses for a turn (system prompt, conversation, retrieved documents, tool definitions and results, and the output) must fit inside it (notebook). Two numbers matter: the advertised limit, a hard API constraint, and the effective limit, the length up to which the model still uses information reliably, which is usually much smaller (RULER).

What counts against the window

flowchart LR
    subgraph IN["Input tokens"]
        A["System prompt"]
        B["Tool definitions"]
        C["Messages: user, assistant, tool results, images, documents"]
        D["Cached prefix (still counts)"]
        E["Kept thinking blocks from earlier turns (model-dependent)"]
    end
    subgraph OUT["Output tokens (bounded by max_tokens)"]
        F["Thinking / reasoning tokens"]
        G["Visible answer"]
    end
    IN --> W["Context window limit"]
    OUT --> W

Per Anthropic’s docs: system prompt, every message (including tool results, images and documents), tool definitions, and the generated output including thinking all count. Prompt caching changes the price of a prefix, not whether it occupies the window. Whether earlier turns’ thinking blocks stay in context depends on the model; newer Claude models keep them (Anthropic). OpenAI reasoning tokens are hidden but likewise occupy the window and are billed as output (OpenAI). See extended thinking and reasoning models.

Limits by vendor (as of 2026-11-14)

Vendor / familyWindowOutput capNotes
Claude Opus 4.6+, Sonnet 4.6+, Haiku 5.5, Fable / Mythos (incl. Mythos Preview)1M tokens, default, no beta header128k per requestStandard pricing except Haiku 5.5 above 100k input (Anthropic)
Older Claude (e.g. Opus 4.5, Haiku 4.5, Sonnet 4.5, deprecated)200kmodel-specific(Anthropic)
Gemini 4 family1,048,576 shared by input, thinking and output65,536 default, more with long decodeIf input + max output exceeds the window, output is capped to what’s left (Gemini)
Gemini 3 / 2.51,048,576 input65,536 output (separate budget)(Gemini)

Tokens are not words: on Claude’s tokenizer since Opus 4.7, 1M tokens is about 555k words, versus about 750k on earlier models (Claude models overview). Non-English text can need several times more tokens for the same content (Petrov et al.). Requests can also hit non-token limits first: Claude accepts up to 600 images or PDF pages per request (100 on 200k models) (Anthropic).

Overflow behaviour

SituationClaudeGemini 4OpenAI reasoning models
Input alone too long400 invalid_request_error, “prompt is too long”Synthesis: request rejectedSynthesis: request rejected
Input + max output > windowClaude 4.5+: accepted; generation stops with stop_reason: model_context_window_exceeded. Earlier models: validation error unless beta headerOutput capped to remaining space—
Output budget exhaustedstop_reason: max_tokensSynthesis: truncatedstatus: incomplete, possibly with no visible text if reasoning used the whole budget

Sources: Anthropic context windows, Messages API, Gemini, OpenAI. OpenAI suggests reserving at least 25,000 tokens for reasoning plus output when starting with reasoning models (OpenAI).

Always check the stop reason. Synthesis: a response cut by max_tokens or the window looks like a normal answer that just ends; code that parses it (e.g. JSON) fails downstream.

Advertised vs effective context

Evidence that models use long contexts unevenly:

  • Position effects. Accuracy is highest when relevant information sits at the start or end of the input and drops significantly in the middle; GPT-3.5-Turbo’s multi-document QA can fall below its no-documents score. Extended-context variants performed about the same as their shorter siblings on inputs both could fit (Lost in the Middle).
  • Needle tests flatter. Models score near-perfectly on vanilla needle-in-a-haystack yet drop sharply on multi-needle, multi-hop and aggregation tasks; only about half of models claiming 32K+ held up at 32K (RULER).
  • Literal matching hides weakness. When the needle shares almost no words with the question, 11 of 13 128K-class models fell below half their short-context score at 32K; GPT-4o went from 99.3% to 69.7% (NoLiMa).
  • Context rot. Across 18 models, performance became increasingly unreliable as input grew, even on simple tasks; distractors compound it; focused ~300-token inputs beat full ~113k-token inputs on LongMemEval (Chroma). Anthropic’s own docs name this context rot (Anthropic).

Why, mechanically: attention spreads a finite budget over n² token pairs, training data contains fewer long sequences, and position-encoding interpolation used to extend windows blurs positional precision, producing gradual decline rather than a cliff (Anthropic engineering). RoPE-based models do not generalise past their trained length without extension methods such as YaRN (YaRN; RoFormer). Standard attention cost also grows quadratically with length, which IO-aware kernels like FlashAttention mitigate but do not remove (FlashAttention). See transformer architecture and long-context degradation.

Managing the window

TechniqueWhat it doesSource
Count firstUse the vendor’s token counter on the exact modelAnthropic token counting
Server-side compactionSummarises earlier turns so a conversation can continue past the limit (beta, Claude 4.6+ and Mythos Preview)Anthropic
Context editingClears old tool results or thinking blocksAnthropic
Just-in-time retrieval, notes, sub-agentsKeep only high-signal tokens in the windowAnthropic engineering
Put key material at the edgesExploits primacy/recencyLost in the Middle

Some Claude models (Sonnet 4.5, 4.6 and 5, Haiku 4.5) receive injected tags telling them their remaining token budget (“context awareness”); Opus 4.7+, Sonnet 5.5, Haiku 5.5 and the Fable/Mythos 5.x models do not, and you can give them an explicit budget with task budgets (beta) instead (Anthropic).

When to use a big window, and when not

Synthesis, grounded in the evidence above:

  • Use long context when the task needs the whole artefact at once (a contract, a codebase slice, a long transcript), when retrieval would fragment cross-references, or for prototyping before building a retrieval pipeline. Pair with prompt caching if the same corpus is queried repeatedly.
  • Prefer retrieval-augmented generation when the corpus exceeds the window, changes often, needs citations, or when cost and latency per query matter: every input token is paid for and processed on every call (notebook).
  • Agents: long-running agent loops fill the window with tool output; budget it explicitly via context engineering and subagents.
  • Evaluate at your real lengths. Vendor needle-in-a-haystack results do not predict multi-hop or aggregation performance (RULER). Build evals at the context sizes you will ship.

Sources

Suggest changes →