Verification and evals
Feedback from outside the model, for the agent at runtime and for the builder over time: tests, hooks, critics and citation checks; capability vs regression evals, pass@k vs pass^k, judge biases and online evals.
Contents
Verification gives an agent a feedback signal that doesn’t come from the model itself, so it can check its work against reality. Evals do the same for the people building the agent.
flowchart LR A["Agent acts"] --> V["Runtime check: tests, hooks, critic, citation pass"] V -->|"fail"| A V -->|"pass"| O["Output / trace"] O --> E["Builder evals: offline suite + online judges"] E -->|"new cases, prompt and tool fixes"| A
Synthesis: the two loops combine the agent-side checks and the builder-side eval flywheel described in the sources below.
For the agent
Models are confident, which is not the same as correct. Effective agents check their output:
- Coding agents run tests, type checkers and linters, and read the errors.
- Browser agents take screenshots of what actually rendered.
- Research agents cite sources and re-read them.
An agent that can run the tests fixes its own bugs. One that can’t ships them confidently. The single most useful verification step is a pass/fail check the agent can run: tests, a build exit code, a linter, a screenshot diff (Claude Code best practices).
Make the check external to the agent’s judgement. An agent stops when work looks done. Claude Code’s escalation ladder runs from the prompt, to /goal (a separate model checks a condition each turn), to a Stop hook that exits with code 2 to send the agent back (capped at eight consecutive blocks), to independent review by a fresh subagent. Ask for evidence (the command and its output), not “all tests pass” (Claude Code best practices). A Stop hook can also return decision: block, and prompt/agent hook types let a model judge the result; note that failing hooks fail open, and onFailure: "block" has no effect on Stop (hooks docs). See agent hooks and middleware and harness engineering.
Test as a user, not just as a unit. In Anthropic’s long-running harness, agents marked features done after unit tests or curl calls; prompting them to test end to end through browser automation substantially improved results. The harness also guards against premature victory with a JSON feature list that agents may only flip from failing to passing (long-running harnesses).
Verify citations separately. Anthropic’s research system ends with a dedicated CitationAgent that maps each claim in the report to its sources (Anthropic research system; see deep research agents).
Constrain, then validate structured output. Instructor validates a reply against a Pydantic model and re-asks with the validation error, up to max_retries, which does not guarantee success (Instructor). The wiki’s recommended pattern is to enforce structure with a strict mode, then validate meaning with one or two bounded retries (see Instructor and Outlines, structured outputs).
Don’t treat the reasoning trace as the check. Chain-of-thought explanations can rationalise answers driven by hidden biasing features without mentioning them (Turpin et al.). Reasoning models mentioned a hint they used only about 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time, and in reward-hacking setups verbalised the hack under 2% of the time. CoT monitoring is useful but not sufficient (Anthropic; see chain-of-thought). Verify the outcome, not the narrative.
For the builder
Evals are a set of realistic tasks with checkable outcomes, run every time a prompt, tool or model changes. Without them, every change is a guess.
- Vocabulary. A task has inputs and success criteria; a trial is one attempt; a grader scores one aspect; the outcome is the final environment state, which can differ from what the agent claims it did (Anthropic evals).
- Capability vs regression. Capability evals should start at a low pass rate; regression evals should sit near 100%, and saturated capability tasks graduate into the regression suite. Start with 20 to 50 tasks drawn from real failures; a 0% pass rate over many trials usually means a broken task (Anthropic evals). Use error analysis to choose them.
- pass@k vs pass^k. pass@k (at least one of k trials succeeds) rises with k; pass^k (all k succeed) falls. At 75% per-trial success, pass^3 is about 42% (Anthropic evals). Because outputs vary run to run, score several trials (see LLM non-determinism and agent evaluation).
- Graders. Code graders are fast and reproducible but brittle; model graders handle nuance but are non-deterministic and need human calibration; humans are the gold standard but slow. Grade outcomes rather than exact paths, and give judges an “Unknown” option. Grader bugs distort scores: Opus 4.5 went from 42% to 95% on CORE-Bench after grading fixes and a less constrained scaffold (Anthropic evals).
- Judge biases. In pairwise judging, GPT-4 gave a consistent verdict when answers were swapped only 65% of the time, and padding attacks fooled weaker judges in over 90% of cases. Mitigation: call the judge twice with the order swapped and count disagreement as a tie; reference answers and few-shot examples also help (Zheng et al.; see LLM-as-judge).
- Rubrics for open-ended work. Anthropic’s research system started with about 20 real queries and one LLM-judge rubric (factual accuracy, citation accuracy, completeness, source quality, tool efficiency). Human testers still caught a preference for SEO content farms (Anthropic research system).
- Evaluate across effort levels. Longer reasoning can lower accuracy on some tasks (distraction, spurious correlations, lost focus on deduction), so evaluate across reasoning lengths, not only at the maximum (inverse scaling; see test-time compute).
- Retrieval separately. For RAG, measure retrieval on its own: Anthropic reported failure rate as 1 - recall@20 on a labelled set (contextual retrieval; see retrieval metrics and RAG).
- Offline gate, online signal. Gate deploys with offline evals, then monitor production with traces and sampled LLM-judge scores on live traffic, implicit feedback and guardrail hits, and feed every failure back into the dataset (notebook; see LLM observability, OpenTelemetry GenAI). On the path to production, an eval score meeting an agreed threshold is the first stage gate (see POC to production).
As architecture
The review-and-critique (generator–critic) pattern builds verification into the system: a separate critic agent checks output against criteria (often an LLM-as-judge), or runs tests, before it’s accepted. Each revision cycle costs another model call. See agent design patterns. Claude Code’s version is review by a fresh subagent that never saw the generator’s reasoning; when asking a reviewer for issues, tell it to skip style, or it will always find something (Claude Code best practices).
Note: the Google Cloud pattern guide gives no evaluation guidance. Builder-side evals originally rested on the agents post alone; they are now covered by Anthropic’s agent evals post and the judge-bias literature cited above.
Related
Agent loop · Subagents · Harness engineering · Agent hooks and middleware
Going deeper
- LLM Evaluation: graders, datasets and statistics for builder-side evals.
- Agent Evaluation: trials, outcome vs trajectory grading, pass@k vs pass^k.
- LLM-as-Judge: model graders and their biases (position, verbosity).
- Error Analysis: reading traces to decide which evals to write.
- Hallucination: runtime checks such as chain-of-verification and existence checks.
- LLM Observability: online evals and traces as the seed for eval datasets.
- Benchmarks: public benchmarks and why they don’t replace task-specific evals.
- LLM Red Teaming: adversarial testing as a complement to capability evals.
- Retrieval metrics: recall@k and ranking metrics for the retrieval half of RAG.
- Chain-of-thought: why visible reasoning is not a faithful verification log.
Sources
- The Anatomy of a Modern AI Agent
- Choose a design pattern for your agentic AI system (Google Cloud)
- Claude Code best practices
- Effective harnesses for long-running agents (Anthropic)
- Hooks reference (Claude Code docs)
- How we built our multi-agent research system (Anthropic)
- Instructor documentation
- Language Models Don’t Always Say What They Think (Turpin et al.)
- Reasoning models don’t always say what they think (Anthropic)
- Demystifying evals for AI agents (Anthropic)
- Judging LLM-as-a-Judge (Zheng et al.)
- Inverse Scaling in Test-Time Compute (Gema et al.)
- LLM evaluation notebook
- Contextual Retrieval (Anthropic)