Coding Agents

LLM agents that read, edit, run and verify code in a real repository through a harness of tools, instructions files and sandboxes (Claude Code, Codex, Cursor, Copilot, Devin).

Contents

A coding agent is an LLM running in an agent loop inside a software repository. It gathers context (search, read files), acts (edit files, run shell commands, use git) and verifies (run tests, builds, linters, screenshots), repeating until the task is done or it hands back to a human. The model supplies the reasoning. The harness around it (Claude Code, OpenAI Codex, Cursor’s agent, GitHub Copilot’s cloud agent, Devin) supplies the tools, decides what enters the context, enforces permissions and sandboxing, and persists state across sessions (How Claude Code works). Benchmark scores and real-world results measure the model and harness together (Anthropic SWE-bench post).

Anatomy of a harness

flowchart LR
  U["Prompt / issue / schedule"] --> H["Harness"]
  I["Instruction files<br/>CLAUDE.md · AGENTS.md · rules"] --> H
  H --> M["Model (reasoning, effort)"]
  M -->|"tool call"| P{"Permission layer<br/>deny/ask rules · hooks · classifier"}
  P -->|"allowed"| S["Sandbox<br/>FS + network isolation"]
  S --> T["Tools: search · read · edit · shell · web · LSP · MCP"]
  T -->|"result (truncated)"| H
  H -->|"checks: tests · Stop hook · reviewer"| V["Verify"]
  V -->|"fail"| M
  V -->|"pass"| O["Diff / commit / PR → human review"]

Synthesis: the diagram merges the Claude Code loop description, its permission/hook/sandbox layering, and the Codex and Copilot equivalents described below.

Core tools. Claude Code groups its built-in tools into file operations, search (glob/regex), execution (shell, tests, git), web, and code intelligence (diagnostics, go-to-definition), plus tools for subagents and asking the user (How it works). Anthropic’s SWE-bench scaffold showed how far a minimal set goes: a bash tool and one view/edit tool reached 49% on SWE-bench Verified with Claude 3.5 Sonnet in January 2025 (SWE-bench post). See tools and function calling.

Tool design is the agent-computer interface (ACI). SWE-agent showed that interface details shift results by several points. On SWE-bench Lite with GPT-4 Turbo (18.0% full configuration), removing the edit command cost 7.7 points, editing without lint guardrails cost 3.0, and showing the full file instead of a 100-line window cost 5.3 (SWE-agent). Lessons repeated across sources:

Design choiceWhySource
Exact-match string replacement (old_str→new_str, error unless exactly one match)Most reliable edit format; failure is explicit and retryableAnthropic
Absolute pathsAgent stops getting lost after cdAnthropic
Lint check rejects broken editsFailed edits cascade: success odds fall from 90.5% to 57.2% after one failed editSWE-agent
Capped, summarised search output (e.g. 50 hits) and truncated tool results (Claude Code default 25k tokens)Keeps context high-signal; truncation message steers to narrower queriesSWE-agent, writing tools
Grep plus semantic (embedding) searchCursor: +12.5% average offline accuracy; in an online A/B, code retention rose most (+2.6%) on 1,000+ file reposCursor
Collapse old observations / compact historyBounded context growthSWE-agent, How it works

Single-model edits replaced the 2024 pattern where a large model wrote edit instructions and a small “apply” model executed them, which was fragile (Cognition).

Context management. Each session starts with a fresh window. Claude Code loads instruction files, auto memory (first 200 lines or 25KB of MEMORY.md), skill descriptions and MCP tool names (definitions are deferred and loaded via tool search). It compacts automatically near the limit, and early detailed instructions can be lost when it does (How it works). As of November 2026 most current Claude models have a 1M-token window, so running out is rare. /clear between unrelated tasks still helps quality and cost because every turn re-sends the whole conversation (best practices). See context engineering, context window and prompt caching.

Instruction files: CLAUDE.md, AGENTS.md, rules

All major agents inject repo-level Markdown (agent instruction files) into the context. These files are guidance the model may weigh, not enforcement (memory docs).

Claude CodeOpenAI CodexCursorCopilot cloud agent
Main fileCLAUDE.md (+ CLAUDE.local.md, managed policy file)AGENTS.md (+ AGENTS.override.md, fallback names).cursor/rules/*.mdc, AGENTS.mdcustom instructions, AGENTS.md (listed as supported by agents.md)
DiscoveryAncestors at launch; subdirectories lazily when touched; concatenated root→cwdGlobal ~/.codex, then root→cwd, at most one file per dir, concatenatedNested AGENTS.md merge, more specific wins; rules by moden/a in sources
Scoping.claude/rules/ with paths: globs; @path imports (max 4 hops)Nesting per directoryAlways / intelligent / glob / manual
Size guidance< 200 lines per file32 KiB combined cap (project_doc_max_bytes); later files dropped< 500 lines per rule
AGENTS.md interopReads AGENTS.md if no CLAUDE.md (v2.1.277+); else import it or change the settingNativeNativeNative

Sources: memory docs, Codex AGENTS.md, Cursor rules, agents.md.

AGENTS.md is a vendor-neutral convention that came out of OpenAI Codex, Amp, Jules, Cursor and Factory. It is now stewarded by the Agentic AI Foundation under the Linux Foundation and reports 60k+ adopting repositories as of November 2026. It has no required fields, the nearest file wins, and an explicit chat prompt overrides everything (agents.md).

What to put in them (best practices): only what the agent can’t discover by reading code, or would re-derive every session. That means exact install/test/typecheck/lint commands (including how to run one test file), misleading areas (“legacy dir is live, don’t copy it”), decisions made outside the repo, and anything you have corrected twice. Leave out architecture descriptions the agent can read for itself. Every line costs attention on every task. Shouting (“IMPORTANT”) tends to make the model over-apply a rule. A line that keeps being ignored means the file is too long, or the rule needs a hook. Long procedures belong in on-demand skills, not always-loaded files.

Permissions and sandboxing

Two orthogonal controls, which both Anthropic and OpenAI separate explicitly:

  1. Permission / approval policy: when the agent must ask. Claude Code offers Manual, Accept edits, Plan (read-only) and Auto modes. Auto is the starting mode for interactive terminal and VS Code sessions (v2.1.283+). In Auto, a classifier model screens risky actions such as curl | bash, production deploys, force pushes, data exfiltration and actions that look driven by content the agent read. Allow/ask/deny rules and PreToolUse hooks run first, and deny rules hold even in bypass mode (best practices). Codex has on-request, never (CI) and granular approval policies, plus an optional auto-review agent that fails closed (Codex security).
  2. Sandbox: what commands can physically do. Claude Code uses bubblewrap (Linux) or seatbelt (macOS) to confine writes to the working directory, and routes network traffic through a domain-allowlisting proxy. It cut permission prompts by 84% internally (sandboxing post). Codex has read-only, workspace-write and danger-full-access modes, network off by default, and .git/.codex/.aws kept read-only inside writable roots (Codex security).

Both layers are needed. Without network isolation an injected agent can exfiltrate keys. Without filesystem isolation it can escape the sandbox (sandboxing post). Permission rules match tool calls, not intent. A Read(.env) deny catches cat .env but not a script that opens the file. A WebFetch domain deny doesn’t stop curl unless the sandbox is on (best practices). See agent guardrails and prompt injection.

Cloud and asynchronous agents move isolation into managed VMs or containers:

  • Claude Code on the web keeps git credentials outside the sandbox. A proxy checks, for example, that a push targets only the configured branch, then attaches the real token (sandboxing post).
  • Codex cloud may use the network during setup, then runs the agent phase offline with secrets removed (Codex security).
  • Copilot’s cloud agent runs either in a GitHub Actions environment (one repo, 59-minute cap) or in a cloud sandbox (can span repos in one org, no custom compute or self-hosted runners), behind a firewall. It pushes only to a copilot/ branch, can’t approve or merge its own PRs (and its requester can’t approve them either), and Actions workflows don’t run until a human with write access approves. It also strips hidden characters from inputs and self-scans with CodeQL and secret scanning (Copilot docs).

Verification: the decisive practice

An agent stops when work looks done, so it needs a check that returns pass/fail (best practices). Claude Code’s escalation ladder runs from the prompt (“run the auth tests and keep going until they pass”), to /goal (a separate model checks a condition each turn), to a Stop hook that exits with code 2 to send the agent back (capped at eight consecutive blocks), to review by a fresh subagent that never saw the reasoning. Ask for evidence (the command and its output), not a claim. When asking a reviewer for issues, tell it to skip style, or it will always find something (best practices). Unit tests can lie by omission: Anthropic’s long-running harness only got reliable feature completion once agents tested as a user through browser automation (see computer-use agents) (long-running harnesses). See verification and evals.

Long-running and parallel work

  • Externalise state. For multi-session builds, an initializer agent writes init.sh, a progress log, a JSON feature list (all failing at first) and an initial commit. Each later session reads these, runs a smoke test, implements one feature and commits. This counters the two observed failures: one-shotting (running out of context mid-feature) and premature victory (long-running harnesses). Related: agent memory.
  • Subagents add parallelism, isolate noisy output and give unbiased review, at a token cost. They don’t make answers smarter (best practices). Cognition (Devin) warns that parallel subagents writing code make conflicting implicit decisions. It prefers a single-threaded agent, with a history-compression model for very long tasks (Cognition). See subagents.
  • Parallel sessions use git worktrees (How it works).
  • Headless runs (claude -p, Codex with approval never) belong in CI and loops. Before running unattended, settle five things: define done, give a self-check, restrict scope, cap turns or spend (--max-turns, --max-budget-usd), and route results to a human-approved PR (best practices).

Benchmarks and what they mean

BenchmarkWhatStatus (2026-11)
SWE-bench (2023)2,294 real issues from 12 Python repos, graded by the fixing PR’s tests. Claude 2 solved 1.96% at launchHistorical baseline (paper)
SWE-bench Verified500 human-validated tasksOpenAI stopped reporting it in Feb 2026: models reproduce gold patches; 59.4% of 138 audited hard tasks had flawed tests; SOTA crept 74.9%→80.9% in six months (OpenAI)
SWE-bench Pro1,865 long-horizon tasks, 41 repos, private held-out and commercial subsetsAt release (v1, Sept 2025) best was GPT-5 at 23.3% public; the v2 paper (Nov 2025) reports up to 43.6% public (Claude Sonnet 4.5) and 17.8% commercial (Claude Opus 4.1). Grading quality disputed by third parties (paper)
Terminal-Bench 2.089 hard terminal tasks with their own environmentsFrontier agents < 65% at publication (paper)

Read scores with care. They mix model and scaffold, hidden tests let agents believe they succeeded when they hadn’t, and some failures are environment problems (SWE-bench post). Productivity is not the same as benchmark score. In METR’s 2025 randomized trial, experienced developers on their own mature repos took 19% longer with early-2025 AI tools (mainly Cursor with Claude 3.5/3.7 Sonnet), yet believed they had been sped up ~20% (METR). See benchmarks and LLM evaluation.

Product landscape (2026-11)

ProductShapeDistinctive points
Claude CodeTerminal-first, also IDE/desktop/web/Slack/CI; local, cloud or remote-control executionHooks, skills, subagents, auto-mode classifier, OS sandbox; default Opus 5.5 at medium effort (How it works, best practices)
OpenAI CodexOpen-source CLI (Apache-2.0) plus IDE extension, desktop app and cloudAGENTS.md origin, sandbox modes × approval policies (Codex security)
CursorIDE-native agentCustom-trained code embeddings plus grep, .mdc rules (Cursor)
GitHub Copilot cloud agentAsynchronous, issue/PR-driven, runs on Actions or GitHub cloud sandboxesInherits GitHub branch protection, reviews and audit (Copilot docs)
Devin (Cognition)Autonomous cloud engineerSingle-threaded, context-sharing philosophy (Cognition)

To build your own coding agent rather than use one, see Claude Agent SDK and OpenAI Agents SDK.

Failure modes

  • Says done when it isn’t: no runnable check, hidden tests, or tests that don’t exercise the real path (best practices, long-running harnesses).
  • Wrong or over-specific fix (~52% of SWE-agent failures) and cascading failed edits (~23%) (SWE-agent).
  • Context poisoning by failed attempts: after two failed corrections, /clear and re-prompt rather than adding a third (best practices).
  • Instruction drift: rules lost at compaction, too-long instruction files, contradictory nested files, or a Codex chain silently capped at 32 KiB (memory docs, Codex AGENTS.md).
  • Approval fatigue: users rubber-stamp prompts, which sandboxes are designed to fix (sandboxing post).
  • Prompt injection from issues, web pages or committed instruction files, contained by sandbox, firewall, hidden-character filtering and branch limits (Codex security, Copilot docs).
  • Cost blowups: hundreds of turns per task, background loops and subagents (SWE-bench post, best practices).

When to use / when not

Synthesis, drawing on the sources above and the Notebook harness tutorial:

  • Good fit: tasks with a crisp pass/fail check (failing test, typecheck, migration across many files, CI fix), repo exploration and Q&A, scaffolding, and batch migrations via headless loops. These are open-ended work where the model should drive but the stop condition is measurable.
  • Use more structure (a graph workflow, narrow tools, human gates) when the path must be guaranteed, as in compliance flows or irreversible operations, and whenever actions touch production systems that checkpoints can’t undo.
  • Be sceptical of claimed speedups for experts on familiar, high-standards codebases. Measure on your own work (METR).
  • FDE checklist: a short AGENTS.md/CLAUDE.md with commands; deny rules plus sandbox for secrets and network; a Stop hook or CI as the definition of done; branch protection and human merge; turn and budget caps for unattended runs.

Sources

Suggest changes →