Source: AI Evals: Everything You Need to Know (Husain and Shankar)

Hamel Husain and Shreya Shankar's long-form FAQ on evals: error analysis via open and axial coding, agent and multi-step evaluation, synthetic data, CI vs production.

Contents
  • What: practitioner FAQ (long blog post), distilled from the authors’ AI evals course
  • Authors: Hamel Husain and Shreya Shankar
  • Date: 18 September 2026 (modified 21 September 2026; earlier versions published from 2025)
  • Original: hamel.dev/blog/posts/evals-faq

Summary

A question-and-answer guide that puts error analysis (reading traces, writing notes, clustering them into a failure taxonomy) at the centre of evaluation work, and derives automated evaluators from the failures found rather than from generic metrics. It has dedicated answers on agents and multi-step pipelines, CI versus production monitoring, eval cadence, and guardrails versus evaluators.

Key claims

  • Error analysis is framed as the most important eval activity. Steps: assemble a dataset of traces, open coding (a human writes free-text notes on what went wrong), axial coding (group the notes into a failure taxonomy and count each category), then iterate.
  • Annotate at least 30 traces personally before leaning on model suggestions; work from a pool of about 100 diverse traces and stop when new traces stop revealing new failure types (theoretical saturation).
  • A single domain expert (a “benevolent dictator”) should own quality judgments in most small and medium organisations.
  • Repeat error analysis on 100+ fresh traces each cycle; the cycles the authors have seen run 2 to 4 weeks, with extra re-runs after significant changes and weekly spot reviews of 10-20 traces.
  • Synthetic inputs: define dimensions (e.g. issue type, customer mood, prior context), hand-write about 20 tuples, generate tuples then turn each into a natural-language query in a separate prompt. Synthetic data cannot tell you how frequent a failure is in production.
  • Prefer binary pass/fail; track gradual progress with several binary sub-checks rather than a Likert scale.
  • The authors report spending 60-80% of development time on error analysis and evaluation; a 100% pass rate may mean the evals are too easy.
  • Agents: first grade end-to-end task success per task, then run step-level diagnostics (tool choice, parameter extraction, error handling, context retention, efficiency in steps/seconds/tokens, goal checkpoints) on the workflows that fail most.
  • Tool calls: check tool name, arguments, result and resulting state as separate checks; verify authorization and preconditions, since a well-formed call can still be the wrong action.
  • Transition failure matrix: rows are the last successful state, columns the state where the first failure occurred; it shows hotspots (a text-to-SQL example: 12 failures at SQL generation to execution, 2 at tool decision to planning).
  • Annotate the first upstream failure in a trace; fixing it often removes downstream errors.
  • Reduce each failure to the simplest reproducing test; use multi-turn tests only when the failure depends on conversation context; sources include simulated users and “N-1 testing” (replaying real conversation prefixes).
  • CI evals: small purpose-built sets (often 100+ examples), mostly deterministic assertions. Production: sample live traces, run reference-free evaluators asynchronously, track confidence intervals. Feed production failures back into CI.
  • Eval cadence depends on cost, saturation and business value; cheap checks on every trace, expensive judges on a nightly sample.
  • Guardrails run inline, must be fast and deterministic, and target clear high-impact failures; evaluators run after the fact and measure nuanced quality.

Feeds

Error analysis · Agent evaluation

Suggest changes →