Coding Agents
LLM agents that read, edit, run and verify code in a real repository through a harness of tools, instructions files and sandboxes (Claude Code, Codex, Cursor, Copilot, Devin).
Contents
A coding agent is an LLM running in an agent loop inside a software repository. It gathers context (search, read files), acts (edit files, run shell commands, use git) and verifies (run tests, builds, linters, screenshots), repeating until the task is done or it hands back to a human. The model supplies the reasoning. The harness around it (Claude Code, OpenAI Codex, Cursor’s agent, GitHub Copilot’s cloud agent, Devin) supplies the tools, decides what enters the context, enforces permissions and sandboxing, and persists state across sessions (How Claude Code works). Benchmark scores and real-world results measure the model and harness together (Anthropic SWE-bench post).
Anatomy of a harness
flowchart LR
U["Prompt / issue / schedule"] --> H["Harness"]
I["Instruction files<br/>CLAUDE.md · AGENTS.md · rules"] --> H
H --> M["Model (reasoning, effort)"]
M -->|"tool call"| P{"Permission layer<br/>deny/ask rules · hooks · classifier"}
P -->|"allowed"| S["Sandbox<br/>FS + network isolation"]
S --> T["Tools: search · read · edit · shell · web · LSP · MCP"]
T -->|"result (truncated)"| H
H -->|"checks: tests · Stop hook · reviewer"| V["Verify"]
V -->|"fail"| M
V -->|"pass"| O["Diff / commit / PR → human review"]Synthesis: the diagram merges the Claude Code loop description, its permission/hook/sandbox layering, and the Codex and Copilot equivalents described below.
Core tools. Claude Code groups its built-in tools into file operations, search (glob/regex), execution (shell, tests, git), web, and code intelligence (diagnostics, go-to-definition), plus tools for subagents and asking the user (How it works). Anthropic’s SWE-bench scaffold showed how far a minimal set goes: a bash tool and one view/edit tool reached 49% on SWE-bench Verified with Claude 3.5 Sonnet in January 2025 (SWE-bench post). See tools and function calling.
Tool design is the agent-computer interface (ACI). SWE-agent showed that interface details shift results by several points. On SWE-bench Lite with GPT-4 Turbo (18.0% full configuration), removing the edit command cost 7.7 points, editing without lint guardrails cost 3.0, and showing the full file instead of a 100-line window cost 5.3 (SWE-agent). Lessons repeated across sources:
| Design choice | Why | Source |
|---|---|---|
Exact-match string replacement (old_str→new_str, error unless exactly one match) | Most reliable edit format; failure is explicit and retryable | Anthropic |
| Absolute paths | Agent stops getting lost after cd | Anthropic |
| Lint check rejects broken edits | Failed edits cascade: success odds fall from 90.5% to 57.2% after one failed edit | SWE-agent |
| Capped, summarised search output (e.g. 50 hits) and truncated tool results (Claude Code default 25k tokens) | Keeps context high-signal; truncation message steers to narrower queries | SWE-agent, writing tools |
| Grep plus semantic (embedding) search | Cursor: +12.5% average offline accuracy; in an online A/B, code retention rose most (+2.6%) on 1,000+ file repos | Cursor |
| Collapse old observations / compact history | Bounded context growth | SWE-agent, How it works |
Single-model edits replaced the 2024 pattern where a large model wrote edit instructions and a small “apply” model executed them, which was fragile (Cognition).
Context management. Each session starts with a fresh window. Claude Code loads instruction files, auto memory (first 200 lines or 25KB of MEMORY.md), skill descriptions and MCP tool names (definitions are deferred and loaded via tool search). It compacts automatically near the limit, and early detailed instructions can be lost when it does (How it works). As of November 2026 most current Claude models have a 1M-token window, so running out is rare. /clear between unrelated tasks still helps quality and cost because every turn re-sends the whole conversation (best practices). See context engineering, context window and prompt caching.
Instruction files: CLAUDE.md, AGENTS.md, rules
All major agents inject repo-level Markdown (agent instruction files) into the context. These files are guidance the model may weigh, not enforcement (memory docs).
| Claude Code | OpenAI Codex | Cursor | Copilot cloud agent | |
|---|---|---|---|---|
| Main file | CLAUDE.md (+ CLAUDE.local.md, managed policy file) | AGENTS.md (+ AGENTS.override.md, fallback names) | .cursor/rules/*.mdc, AGENTS.md | custom instructions, AGENTS.md (listed as supported by agents.md) |
| Discovery | Ancestors at launch; subdirectories lazily when touched; concatenated root→cwd | Global ~/.codex, then root→cwd, at most one file per dir, concatenated | Nested AGENTS.md merge, more specific wins; rules by mode | n/a in sources |
| Scoping | .claude/rules/ with paths: globs; @path imports (max 4 hops) | Nesting per directory | Always / intelligent / glob / manual | |
| Size guidance | < 200 lines per file | 32 KiB combined cap (project_doc_max_bytes); later files dropped | < 500 lines per rule | |
| AGENTS.md interop | Reads AGENTS.md if no CLAUDE.md (v2.1.277+); else import it or change the setting | Native | Native | Native |
Sources: memory docs, Codex AGENTS.md, Cursor rules, agents.md.
AGENTS.md is a vendor-neutral convention that came out of OpenAI Codex, Amp, Jules, Cursor and Factory. It is now stewarded by the Agentic AI Foundation under the Linux Foundation and reports 60k+ adopting repositories as of November 2026. It has no required fields, the nearest file wins, and an explicit chat prompt overrides everything (agents.md).
What to put in them (best practices): only what the agent can’t discover by reading code, or would re-derive every session. That means exact install/test/typecheck/lint commands (including how to run one test file), misleading areas (“legacy dir is live, don’t copy it”), decisions made outside the repo, and anything you have corrected twice. Leave out architecture descriptions the agent can read for itself. Every line costs attention on every task. Shouting (“IMPORTANT”) tends to make the model over-apply a rule. A line that keeps being ignored means the file is too long, or the rule needs a hook. Long procedures belong in on-demand skills, not always-loaded files.
Permissions and sandboxing
Two orthogonal controls, which both Anthropic and OpenAI separate explicitly:
- Permission / approval policy: when the agent must ask. Claude Code offers Manual, Accept edits, Plan (read-only) and Auto modes. Auto is the starting mode for interactive terminal and VS Code sessions (v2.1.283+). In Auto, a classifier model screens risky actions such as
curl | bash, production deploys, force pushes, data exfiltration and actions that look driven by content the agent read. Allow/ask/deny rules and PreToolUse hooks run first, and deny rules hold even in bypass mode (best practices). Codex hason-request,never(CI) andgranularapproval policies, plus an optional auto-review agent that fails closed (Codex security). - Sandbox: what commands can physically do. Claude Code uses bubblewrap (Linux) or seatbelt (macOS) to confine writes to the working directory, and routes network traffic through a domain-allowlisting proxy. It cut permission prompts by 84% internally (sandboxing post). Codex has
read-only,workspace-writeanddanger-full-accessmodes, network off by default, and.git/.codex/.awskept read-only inside writable roots (Codex security).
Both layers are needed. Without network isolation an injected agent can exfiltrate keys. Without filesystem isolation it can escape the sandbox (sandboxing post). Permission rules match tool calls, not intent. A Read(.env) deny catches cat .env but not a script that opens the file. A WebFetch domain deny doesn’t stop curl unless the sandbox is on (best practices). See agent guardrails and prompt injection.
Cloud and asynchronous agents move isolation into managed VMs or containers:
- Claude Code on the web keeps git credentials outside the sandbox. A proxy checks, for example, that a push targets only the configured branch, then attaches the real token (sandboxing post).
- Codex cloud may use the network during setup, then runs the agent phase offline with secrets removed (Codex security).
- Copilot’s cloud agent runs either in a GitHub Actions environment (one repo, 59-minute cap) or in a cloud sandbox (can span repos in one org, no custom compute or self-hosted runners), behind a firewall. It pushes only to a
copilot/branch, can’t approve or merge its own PRs (and its requester can’t approve them either), and Actions workflows don’t run until a human with write access approves. It also strips hidden characters from inputs and self-scans with CodeQL and secret scanning (Copilot docs).
Verification: the decisive practice
An agent stops when work looks done, so it needs a check that returns pass/fail (best practices). Claude Code’s escalation ladder runs from the prompt (“run the auth tests and keep going until they pass”), to /goal (a separate model checks a condition each turn), to a Stop hook that exits with code 2 to send the agent back (capped at eight consecutive blocks), to review by a fresh subagent that never saw the reasoning. Ask for evidence (the command and its output), not a claim. When asking a reviewer for issues, tell it to skip style, or it will always find something (best practices). Unit tests can lie by omission: Anthropic’s long-running harness only got reliable feature completion once agents tested as a user through browser automation (see computer-use agents) (long-running harnesses). See verification and evals.
Long-running and parallel work
- Externalise state. For multi-session builds, an initializer agent writes
init.sh, a progress log, a JSON feature list (all failing at first) and an initial commit. Each later session reads these, runs a smoke test, implements one feature and commits. This counters the two observed failures: one-shotting (running out of context mid-feature) and premature victory (long-running harnesses). Related: agent memory. - Subagents add parallelism, isolate noisy output and give unbiased review, at a token cost. They don’t make answers smarter (best practices). Cognition (Devin) warns that parallel subagents writing code make conflicting implicit decisions. It prefers a single-threaded agent, with a history-compression model for very long tasks (Cognition). See subagents.
- Parallel sessions use git worktrees (How it works).
- Headless runs (
claude -p, Codex with approvalnever) belong in CI and loops. Before running unattended, settle five things: define done, give a self-check, restrict scope, cap turns or spend (--max-turns,--max-budget-usd), and route results to a human-approved PR (best practices).
Benchmarks and what they mean
| Benchmark | What | Status (2026-11) |
|---|---|---|
| SWE-bench (2023) | 2,294 real issues from 12 Python repos, graded by the fixing PR’s tests. Claude 2 solved 1.96% at launch | Historical baseline (paper) |
| SWE-bench Verified | 500 human-validated tasks | OpenAI stopped reporting it in Feb 2026: models reproduce gold patches; 59.4% of 138 audited hard tasks had flawed tests; SOTA crept 74.9%→80.9% in six months (OpenAI) |
| SWE-bench Pro | 1,865 long-horizon tasks, 41 repos, private held-out and commercial subsets | At release (v1, Sept 2025) best was GPT-5 at 23.3% public; the v2 paper (Nov 2025) reports up to 43.6% public (Claude Sonnet 4.5) and 17.8% commercial (Claude Opus 4.1). Grading quality disputed by third parties (paper) |
| Terminal-Bench 2.0 | 89 hard terminal tasks with their own environments | Frontier agents < 65% at publication (paper) |
Read scores with care. They mix model and scaffold, hidden tests let agents believe they succeeded when they hadn’t, and some failures are environment problems (SWE-bench post). Productivity is not the same as benchmark score. In METR’s 2025 randomized trial, experienced developers on their own mature repos took 19% longer with early-2025 AI tools (mainly Cursor with Claude 3.5/3.7 Sonnet), yet believed they had been sped up ~20% (METR). See benchmarks and LLM evaluation.
Product landscape (2026-11)
| Product | Shape | Distinctive points |
|---|---|---|
| Claude Code | Terminal-first, also IDE/desktop/web/Slack/CI; local, cloud or remote-control execution | Hooks, skills, subagents, auto-mode classifier, OS sandbox; default Opus 5.5 at medium effort (How it works, best practices) |
| OpenAI Codex | Open-source CLI (Apache-2.0) plus IDE extension, desktop app and cloud | AGENTS.md origin, sandbox modes × approval policies (Codex security) |
| Cursor | IDE-native agent | Custom-trained code embeddings plus grep, .mdc rules (Cursor) |
| GitHub Copilot cloud agent | Asynchronous, issue/PR-driven, runs on Actions or GitHub cloud sandboxes | Inherits GitHub branch protection, reviews and audit (Copilot docs) |
| Devin (Cognition) | Autonomous cloud engineer | Single-threaded, context-sharing philosophy (Cognition) |
To build your own coding agent rather than use one, see Claude Agent SDK and OpenAI Agents SDK.
Failure modes
- Says done when it isn’t: no runnable check, hidden tests, or tests that don’t exercise the real path (best practices, long-running harnesses).
- Wrong or over-specific fix (~52% of SWE-agent failures) and cascading failed edits (~23%) (SWE-agent).
- Context poisoning by failed attempts: after two failed corrections,
/clearand re-prompt rather than adding a third (best practices). - Instruction drift: rules lost at compaction, too-long instruction files, contradictory nested files, or a Codex chain silently capped at 32 KiB (memory docs, Codex AGENTS.md).
- Approval fatigue: users rubber-stamp prompts, which sandboxes are designed to fix (sandboxing post).
- Prompt injection from issues, web pages or committed instruction files, contained by sandbox, firewall, hidden-character filtering and branch limits (Codex security, Copilot docs).
- Cost blowups: hundreds of turns per task, background loops and subagents (SWE-bench post, best practices).
When to use / when not
Synthesis, drawing on the sources above and the Notebook harness tutorial:
- Good fit: tasks with a crisp pass/fail check (failing test, typecheck, migration across many files, CI fix), repo exploration and Q&A, scaffolding, and batch migrations via headless loops. These are open-ended work where the model should drive but the stop condition is measurable.
- Use more structure (a graph workflow, narrow tools, human gates) when the path must be guaranteed, as in compliance flows or irreversible operations, and whenever actions touch production systems that checkpoints can’t undo.
- Be sceptical of claimed speedups for experts on familiar, high-standards codebases. Measure on your own work (METR).
- FDE checklist: a short AGENTS.md/CLAUDE.md with commands; deny rules plus sandbox for secrets and network; a Stop hook or CI as the definition of done; branch protection and human merge; turn and budget caps for unattended runs.
Sources
- How Claude Code works
- Best practices for Claude Code
- How Claude remembers your project
- Claude Code sandboxing (Anthropic, 2025)
- Raising the bar on SWE-bench Verified (Anthropic, 2025)
- Effective harnesses for long-running agents (Anthropic, 2025)
- Writing effective tools for agents (Anthropic, 2025)
- AGENTS.md
- Codex: custom instructions with AGENTS.md
- Codex: agent approvals and security
- GitHub Copilot cloud agent docs
- Cursor: improving agent with semantic search
- Cursor rules docs
- Don’t Build Multi-Agents (Cognition, 2025)
- SWE-bench (Jimenez et al., 2023)
- SWE-agent (Yang et al., 2024)
- Why we no longer evaluate SWE-bench Verified (OpenAI, 2026)
- SWE-Bench Pro (Scale AI, 2025)
- Terminal-Bench (2026)
- METR developer productivity RCT (2025)
- Harness vs Loop vs Graph Engineer (Notebook)