Agent Evaluation
Evaluating multi-step agents rather than single completions: outcome vs trajectory grading, tool-call checks, simulated environments and users, pass@k vs pass^k, and cost and turn budgets.
Contents
Agent evaluation measures whether an agent completes a task across many model calls, tool calls and (often) conversation turns, in an environment whose state it changes. It differs from grading one completion in three ways: the unit of evaluation is a whole trial (one attempt, producing a transcript and a final environment state), success is judged on what the agent did rather than what it said, and because each trial is long, stochastic and expensive, reliability and cost become first-class metrics (Anthropic). General eval mechanics (graders, datasets, statistics) live on LLM evaluation; this page covers what changes for agents.
Vocabulary
Anthropic’s terms, now widely used (source):
| Term | Meaning |
|---|---|
| Task | Inputs plus success criteria |
| Trial | One attempt at a task; run several because outputs vary |
| Transcript / trace | Every message, tool call and result in the trial |
| Outcome | Final environment state, which can differ from what the agent claims |
| Grader | Logic that scores one aspect (code, model or human) |
| Eval harness | Runs tasks, isolates environments, records and grades; distinct from the agent harness |
Outcome vs trajectory grading
flowchart LR T["Task + initial state"] --> A["Agent harness<br/>(model, tools, loop)"] U["Simulated user<br/>(optional)"] <--> A A --> TR["Transcript:<br/>messages + tool calls"] A --> S["Final environment state"] S --> OG["Outcome graders:<br/>state diff, tests, assertions"] TR --> PG["Trajectory graders:<br/>tool-call match, LLM judge, step counts"] OG --> R["Score per trial"] PG --> R R --> K["Aggregate over k trials:<br/>pass@k, pass^k, cost, turns"]
Outcome grading checks the end state: the database after a τ-bench conversation equals an annotated goal state (Yao et al.); a SWE-bench patch makes the issue’s failing tests pass without breaking others (harness). It is objective and tolerant of any valid path, which is why Anthropic advises grading outcomes rather than exact paths (source).
Trajectory grading inspects the path: which tools were called, with what arguments, in what order, and how efficiently. It catches things outcomes miss. τ-bench itself notes that a reward of 1 is necessary but not sufficient, since an agent can reach the right state while skipping a required user confirmation (Yao et al.). It also localises failures: Husain and Shankar recommend grading end-to-end success first, then running step-level diagnostics (tool choice, parameter extraction, error handling, context retention, efficiency) on the workflows that fail most (FAQ).
The trade-off: strict trajectory matching penalises valid alternative paths, correct error recovery and harmless redundant calls (BFCL V3). Practical trajectory checks are therefore relaxed:
| Matcher | What passes | Tooling |
|---|---|---|
| Strict | Same tool calls, same order | agentevals strict; ADK tool_trajectory_avg_score with its default EXACT match type (default threshold 1.0) |
| Unordered | Same calls, any order | agentevals unordered |
| Superset | Required calls present; extras allowed | agentevals superset; ADK tool_trajectory_avg_score with match_type ANY_ORDER (or IN_ORDER to also keep the expected order); BFCL “minimal viable execution path” |
| Subset | No calls outside an allow-list | agentevals subset |
| LLM judge | Rubric over the whole trajectory | create_trajectory_llm_as_judge; ADK rubric_based_tool_use_quality_v1 |
Sources: agentevals, ADK, BFCL. Promptfoo offers similar trajectory:tool-used, trajectory:tool-sequence, trajectory:step-count and tool-call-f1 assertions (source). agentevals also lets argument matching be set per tool (exact, ignore, a field list, or a custom comparator), which is how you tolerate e.g. case differences in a city argument (source).
Synthesis: grade the outcome as the pass/fail signal, and use relaxed trajectory checks (superset of required calls, forbidden calls, step caps) as additional binary assertions, not as the primary score.
Tool-call correctness
A tool call can fail at several independent layers, so check them separately (FAQ): selection (right tool, or correctly no tool), arguments (extracted correctly, not guessed), result handling (empty results, API errors), resulting state, and authorisation and preconditions, since a syntactically valid call can still be the wrong action. OpenAI’s guidance likewise adds tool-selection and argument-precision evals for agents and handoff accuracy for multi-agent systems (source).
BFCL shows how to grade this at scale (source): V1 parsed single calls into an AST and matched parameter values against acceptable answers; V3’s multi-turn tasks combine a state check after every turn (verifies writes) with a subset-matched response check (the required read calls must appear, which catches the model guessing facts it should have looked up). Its categories double as a checklist of behaviours to test: missing parameters (should ask), missing functions (should say it cannot), long context, and composites.
Environment and simulation-based evals
Agents need an environment to act on. Three common patterns:
- Simulated user + stateful tools (τ-bench). An LLM plays the customer from a hidden instruction; the agent gets a policy document and Python tools over JSON databases. The original setup ran the user at temperature 1.0, the agent at 0.0, capped trials at 30 agent actions, and used 115 retail and 50 airline tasks; reward requires both the correct final database and required information in the agent’s replies (Yao et al.). τ²-bench adds dual control: the user also has tools (toggling phone settings in a telecom domain), and success criteria widen to state assertions, natural-language assertions and action matching. Agents score markedly lower in dual control than when they act alone (Barres et al.).
- Containerised repositories (SWE-bench, Terminal-Bench). Each task ships an environment and tests. SWE-bench marks an instance resolved only if the patch applies and all FAIL_TO_PASS and PASS_TO_PASS tests pass, run in per-instance Docker images (harness, paper); Terminal-Bench 2.0 gives each of 89 tasks its own environment, reference solution and tests, run through the Harbor harness (source).
- Framework-native simulation. ADK evalsets can define a scenario for an LLM user simulator; in that mode the reference trajectory and response-match metrics are unavailable, because the conversation is not fixed (ADK).
Simulators are themselves a source of error. τ²-bench audited its user simulator and found critical (outcome-affecting) errors in roughly 6-13% of conversations depending on domain (Barres et al.); the original τ-bench traced 4 of 40 sampled gpt-4o retail failures to typos or ambiguity in the user instructions (a task bug, then fixed) rather than to the agent (Yao et al.). Start every trial from a clean environment to avoid correlated failures and leakage between trials (Anthropic).
pass@k vs pass^k
Both are estimated from n trials per task with c successes (Chen et al., Yao et al.):
| Metric | Question | Unbiased estimator (per task, averaged) | Behaviour as k grows |
|---|---|---|---|
| pass@k | Does at least one of k attempts succeed? | 1 − C(n−c, k) / C(n, k) | Rises |
| pass^k | Do all k attempts succeed? | C(c, k) / C(n, k) | Falls |
pass@k fits settings with a cheap verifier and retries (generate several patches, keep one that passes tests). pass^k fits customer-facing agents that must work every time: at 75% per-trial success, pass^3 is about 42% (Anthropic); gpt-4o’s τ-retail pass^1 of 61.2% fell below 25% at pass^8 (Yao et al.). Use the combinatorial estimators, not 1 − (1 − p̂)^k, which is biased, and draw n ≥ k samples (Chen et al.). For pass@k, higher temperature helps large k (Chen et al.); do not lower temperature just to reduce variance, since it can shift the expected score (Miller). Synthesis: if per-trial outcomes were independent with rate p, pass^k would be p^k; real failures cluster on hard tasks, so measure pass^k directly rather than extrapolating.
Cost, turn and time budgets
Accuracy alone rewards waste. On HumanEval, a retry-with-rising-temperature baseline matched complex agents: 93.2% for $2.45 versus LATS at 88.0% for $134.50 (Kapoor et al. 2024). Report dollar cost plus token counts (so others can reprice), plot accuracy against cost, and compare on the Pareto frontier. Budgets also shape scores directly: BFCL force-fails turns over 20 steps (source), τ-bench caps 30 actions (source), and ADK reports tool calls, inference calls, tokens and duration as non-gating metrics (source). More thinking is not free accuracy: HAL found higher reasoning effort reduced accuracy in most of its runs (Kapoor et al. 2025), while on τ-bench airline a prompted think tool raised pass^1 from 0.370 to 0.570 (Anthropic). Expense limits rigour too: at SWE-agent’s $4/task cap, one run over the 2,000+ SWE-bench tasks could cost over $8,000, which discourages repeated trials and error bars (Kapoor et al. 2024).
Failure modes of agent evals
- Broken tasks and graders. A 0% pass rate across many trials usually means a broken task; grading fixes plus a less constrained scaffold moved Opus 4.5 from 42% to 95% on CORE-Bench (Anthropic). OpenAI’s audit found 59.4% of hard SWE-bench Verified problems had flawed tests or descriptions (source).
- Contamination and saturation. Frontier models could reproduce SWE-bench Verified gold patches, and OpenAI stopped reporting it in 2026 (source).
- Shortcuts and missing holdouts. Agents hard-coded site-specific policies on WebArena; most of 17 surveyed agent benchmarks lacked suitable holdouts (Kapoor et al. 2024).
- Reward hacking visible only in logs. HAL found an agent searching for the benchmark on Hugging Face and agents misusing credit cards while booking flights (Kapoor et al. 2025). Read transcripts, not only scores.
- Harness state bugs. SWE-bench caches results by
run_idand instance, so re-running a changed patch under the same id silently reuses the old result (harness). - Compounding errors. In SWE-agent, success probability fell from 90.5% to 57.2% after one failed edit (Yang et al.); annotate the first upstream failure (FAQ).
- Multi-agent specific failures. Step repetition (15.7%), reasoning-action mismatch (13.2%), unawareness of termination conditions (12.4%) and incorrect or missing verification are among the most common in MAST’s 1,642 labelled traces (MAST).
Practical guidance
- Derive tasks from real failures via error analysis; 20-50 tasks are enough to start, because early changes have large effects (Anthropic).
- Make each task unambiguous: two experts should agree on the verdict. Include negative cases (should not act / should ask) (Anthropic; BFCL).
- Reproduce failures in the simplest form; use multi-turn or simulated-user tests only when the failure depends on conversation context; replay real conversation prefixes (“N-1 testing”) (FAQ).
- Keep capability evals (low pass rate, hill-climbing) separate from regression evals (near 100%, run on every change); graduate saturated tasks (Anthropic).
- Run several trials per task, report pass^k for user-facing agents, with standard errors and paired comparisons (Miller).
- Track cost, turns and latency next to success; gate on success, budget-alert on the rest (Kapoor et al. 2024).
- Prefer deterministic graders (state diffs, tests, call assertions) in CI; use calibrated LLM judges for trajectory quality and on sampled production traces (FAQ).
When not to build a bespoke agent environment: if the agent is effectively a fixed workflow, test each step like a single completion. Public benchmarks are useful for model selection but do not replace task suites built from your own traces. See also LLM observability, coding agents, verification and evals.
Sources
- Demystifying evals for AI agents (Anthropic)
- AI Evals FAQ (Husain and Shankar)
- τ-bench (Yao et al.) · τ²-bench (Barres et al.)
- Codex / HumanEval and pass@k (Chen et al.)
- SWE-bench harness docs · SWE-bench paper · SWE-bench Verified retired (OpenAI) · Terminal-Bench · SWE-agent
- BFCL V3 · agentevals · ADK evaluation · Promptfoo assertions
- AI Agents That Matter · Holistic Agent Leaderboard
- MAST (Cemri et al.) · Think tool (Anthropic)
- Adding error bars to evals (Miller) · OpenAI evaluation best practices