Source: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker et al., 2025)
METR randomized controlled trial: experienced OSS developers using early-2025 AI tools (mainly Cursor + Claude 3.5/3.7 Sonnet) took 19% longer, while believing they were faster.
Contents
- What: research paper (randomized controlled trial)
- Authors: Joel Becker, Nate Rush, Elizabeth Barnes, David Rein (METR)
- Date: 2025-07-12 (v2 2025-07-25)
- Original: arXiv:2507.09089
Summary
16 experienced developers completed 246 real tasks on mature projects they knew well (about five years’ experience on average); each task was randomly assigned to allow or forbid AI tools.
Key claims
- Allowing AI increased completion time by about 19%.
- Developers forecast a 24% speedup beforehand and still believed afterwards that AI had saved about 20%; economics and ML experts forecast 38 to 39% speedups.
- The authors examined 20 possible contributing factors (e.g. repository size and quality bar, tool familiarity) and judge the slowdown unlikely to be mainly an artefact of the design.
- Small sample and early-2025 tools; results may not generalise to other settings or later agents.