Context engineering
Deciding what goes into an agent's finite context window, and when.
Contents
Context engineering is the practice of controlling what is in an agent’s context window at each step: system prompt, tool definitions, conversation history and tool results. The window is finite, and models use information less well as it fills. Anthropic frames context as an “attention budget” that every token spends, with diminishing returns (Anthropic). The anatomy post calls managing it arguably the most important skill in building agents (anatomy).
Why it matters
- Context rot. Holding task difficulty fixed and varying only input length, all 18 models Chroma tested degraded. On LongMemEval every model scored much higher on a focused ~300-token prompt than on the full ~113k-token one. Distractors make it worse (Chroma).
- Position matters. Accuracy follows a U-shape: information at the start or end is used far better than information in the middle (Liu et al.).
- Mechanism. Self-attention relates every token pair, so attention stretches thin as n grows, and training data has fewer long sequences than short ones (Anthropic).
The goal is the smallest set of high-signal tokens that gets the task done, not the largest window (Anthropic).
Techniques
Synthesis: the levers below, by where they act in the agent loop.
flowchart LR
S["Static prefix: system prompt, tools, instruction files, skill metadata"] --> W["Context window"]
R["Just-in-time retrieval (search / read tools)"] --> W
T["Tool results (lean, paginated, truncated)"] --> W
W --> C{"Near the limit?"}
C -- "yes" --> K["Compact / clear old tool results"]
K --> W
W --> N["Notes in files / memory"]
N -. "reload after reset" .-> W
W --> D["Subagent: fresh window, returns summary"]
D --> W- Load on demand. Give the agent search and read tools and lightweight references (paths, queries) rather than pre-loading documents (Anthropic). This is how coding agents work in codebases far larger than their window; agentic RAG is the retrieval form of the same idea.
- Keep tool output lean. Summarise, paginate, filter and set default limits; offer concise vs detailed response formats. Claude Code caps tool responses at 25,000 tokens by default, and truncation messages should steer the agent towards narrower queries (Anthropic).
- Compact long sessions. Near the limit, replace older turns with a summary that keeps decisions and open threads (context compaction). Providers now do this server-side: Anthropic’s compaction runs on demand or at a token threshold and recommends on-demand where available (Anthropic); OpenAI’s Responses API compacts past a
compact_thresholdinto an encrypted, non-readable item (OpenAI). - Clear stale tool results. Context editing is the rule-based alternative to summarising. Anthropic’s
clear_tool_uses_20250919clears the oldest tool results first, by default once input passes 100,000 tokens, keeping the last 3 tool uses;clear_thinking_20251015does the same for thinking blocks (Anthropic). LangChain’sContextEditingMiddlewareuses the same defaults, andSummarizationMiddlewarekeeps the last 20 messages (LangChain; see agent hooks and middleware). - Write things down. Notes in files (plans, progress logs, to-do lists) let an agent recover its state after a reset or compaction (see agent memory). Anthropic’s context editing warns the model before clearing so it can save results to memory files first (Anthropic). For multi-session builds, compaction alone was not enough; a harness that kept state in a progress log, a JSON feature list and git was (Anthropic).
- Delegate. Subagents do context-heavy work in a fresh window and return only a summary (Anthropic). Cognition argues against parallel agents that cannot see each other’s decisions; it prefers one linear agent, plus a model that compresses history into key decisions for very long tasks (Cognition).
In multi-agent systems the same principles apply per agent: isolate each agent’s context, persist what later steps need, and compress large data before passing it on (Google Cloud; see agent handoffs).
Hidden costs in the window
- Instruction files. Claude Code loads CLAUDE.md files from ancestor directories at launch and from subdirectories lazily, concatenated root-down, and advises under 200 lines per file (Claude Code). Codex stops adding AGENTS.md files at 32 KiB combined by default (Codex). One study found such files did not generally raise coding-agent success and added over 20% inference cost, because agents obeyed unnecessary requirements (Gloaguen et al.). See agent instruction files.
- Skills. Progressive disclosure keeps the fixed cost small: about 100 tokens of metadata per skill always loaded, under 5k tokens of instructions when triggered, and bundled resources free until read (Anthropic; see agent skills).
- Thinking. On keep-all Claude models (Opus 4.5+, Sonnet 4.6+, Haiku 5.5, Fable/Mythos), earlier turns’ thinking stays in context and is billed as input (Anthropic; see extended thinking).
- Screenshots. For computer-use agents each screenshot costs about 1,000-1,800 input tokens. Anthropic suggests pruning old screenshots in batches (e.g. keep the last 3, prune every 25 turns) to preserve caching, but on Claude 5.5 and Fable/Mythos 5.1-5.2 pruning invalidates every later thinking block (Anthropic).
Interaction with prompt caching
Prompt caching needs a byte-identical prefix in the order tools → system → messages, so putting stable content first and appending, never editing, is also what makes context cacheable (Anthropic). Context engineering and caching pull against each other: changing tool definitions invalidates the whole cache (Anthropic), and clearing tool results invalidates the cached prefix, which is why clear_at_least exists to make each clear free enough tokens to justify the new cache write (Anthropic).
Related
Agent loop · Context compaction · Harness engineering · Agent hooks and middleware · Agent handoffs
Going deeper
- Context window: current window sizes per model and provider-side compaction / context editing.
- Context compaction: server-side and client-side summarisation in detail.
- Long-context degradation: the evidence (lost-in-the-middle, context rot) behind keeping context small.
- Agent Memory: compaction, tool-result clearing and long-term memory stores.
- Prompt Caching: why stable-first context ordering is also what makes it cacheable.
- Agentic RAG: just-in-time retrieval as the ’load on demand’ technique.
- Agent Instruction Files: how CLAUDE.md / AGENTS.md files are loaded into context.
- Agent Skills: progressive disclosure: load instructions only when needed.
- Harness engineering: externalising state across sessions for long-running agents.
- Prompt engineering: writing the static parts of the context well.
Sources
- The Anatomy of a Modern AI Agent
- Choose a design pattern for your agentic AI system (Google Cloud)
- Effective context engineering for AI agents (Anthropic)
- Context Rot (Chroma)
- Lost in the Middle (Liu et al.)
- Writing effective tools for agents (Anthropic)
- Compaction overview (Claude API docs)
- Context editing (Claude API docs)
- Compaction (OpenAI guide)
- LangChain agents, middleware and structured output docs
- Prompt caching (Claude API docs)
- Claude Code memory docs
- Custom instructions with AGENTS.md (OpenAI Codex docs)
- Evaluating AGENTS.md (Gloaguen et al., 2026)
- Agent Skills overview (Claude Platform docs)
- Thinking (Claude API docs)
- Computer use tool (Claude API docs)
- Effective harnesses for long-running agents (Anthropic)
- Don’t Build Multi-Agents (Cognition)