Source: Terminal-Bench (Merrill et al., 2026)
Paper introducing Terminal-Bench 2.0: 89 hard, realistic command-line tasks with per-task environments and verification tests.
Contents
- What: research paper (ICLR 2026)
- Authors: Mike A. Merrill, Ludwig Schmidt and 83 others
- Date: 2026-01-17
- Original: arXiv:2601.11868
Summary
Measures agents on end-to-end terminal work rather than patch generation: each task ships its own environment, a human-written reference solution and tests. Tasks are inspired by real workflows (e.g. training a model, building software from source, reverse engineering).
Key claims
- Terminal-Bench 2.0 has 89 curated hard tasks.
- At publication, frontier models and agents scored below 65%; smaller models around 15%.
- Includes an error analysis of agent failure modes; tasks run through the Harbor harness.