Source: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker et al., 2025)

METR randomized controlled trial: experienced OSS developers using early-2025 AI tools (mainly Cursor + Claude 3.5/3.7 Sonnet) took 19% longer, while believing they were faster.

Contents
  • What: research paper (randomized controlled trial)
  • Authors: Joel Becker, Nate Rush, Elizabeth Barnes, David Rein (METR)
  • Date: 2025-07-12 (v2 2025-07-25)
  • Original: arXiv:2507.09089

Summary

16 experienced developers completed 246 real tasks on mature projects they knew well (about five years’ experience on average); each task was randomly assigned to allow or forbid AI tools.

Key claims

  • Allowing AI increased completion time by about 19%.
  • Developers forecast a 24% speedup beforehand and still believed afterwards that AI had saved about 20%; economics and ML experts forecast 38 to 39% speedups.
  • The authors examined 20 possible contributing factors (e.g. repository size and quality bar, tool familiarity) and judge the slowdown unlikely to be mainly an artefact of the design.
  • Small sample and early-2025 tools; results may not generalise to other settings or later agents.

Feeds

Coding agents

Suggest changes →