Source: AI Agents That Matter (Kapoor et al., 2024)

Princeton paper arguing agent benchmarks must report cost alongside accuracy, use proper holdouts, and be reproducible; simple baselines rival complex agents.

Contents
  • What: research paper, arXiv:2407.01502
  • Authors: Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan
  • Date: 1 July 2024
  • Original: arxiv.org/abs/2407.01502

Summary

Critiques agent evaluation practice: accuracy-only leaderboards reward needlessly expensive agents, benchmarks lack holdouts so agents overfit with shortcuts, and results are often irreproducible. It proposes cost-accuracy Pareto reporting, separating model evaluation from downstream evaluation, and holdouts matched to the claimed generality.

Key claims

  • On HumanEval (five runs, 164 problems), simple baselines matched complex agents: “warming” (retries with rising temperature, GPT-4) 93.2% for $2.45 versus LATS (GPT-4) 88.0% for $134.50; Reflexion 87.8% for $3.90; an escalation baseline 85.0% for $0.27.
  • Similar accuracy can come at costs differing by nearly two orders of magnitude, so accuracy must be plotted against cost.
  • Joint cost-accuracy optimisation (DSPy + Optuna on HotPotQA) cut variable cost 53% (GPT-3.5) and 41% (Llama-3-70B) at similar accuracy.
  • Holdouts should match the claimed generality (in-distribution samples, out-of-distribution samples, unseen tasks, unseen domains); of 17 benchmarks surveyed, only 5 had appropriate holdouts and seven had none (nor plans for one).
  • WebArena example: STeP (35.8%) hard-coded site-specific policies and more than doubled the paper’s best baseline, which would not survive site changes.
  • Reproducibility problems: missing test cases, reported scores above the authors’ own reruns, harness bugs that marked wrong answers correct, and order-dependent results from rate limits. At SWE-agent’s $4/task cap, one run over SWE-bench’s 2,000+ tasks could cost over $8,000, discouraging repeated runs and error bars.
  • Recommendations: report dollar cost plus token counts, publish standard eval scripts, include error bars.

Feeds

Agent evaluation

Suggest changes →