Source: Why we no longer evaluate SWE-bench Verified (OpenAI, 2026)

OpenAI's February 2026 post dropping SWE-bench Verified because of training-data contamination and flawed tests, recommending SWE-bench Pro instead.

Contents
  • What: research blog post
  • Publisher: OpenAI
  • Date: 2026-02-23
  • Original: Why we no longer evaluate SWE-bench Verified
  • Note: the original refused automated fetching; on 2026-11-14 the 138-problem audit, 59.4%, o3/64-run selection, six-reviewer, contamination and 74.9%→80.9% figures were re-checked against a reader-mode copy of the post. The exact publication day was not re-confirmed (February 2026).

Summary

SWE-bench Verified (a 500-task human-validated subset OpenAI released in 2024) is judged to no longer measure frontier coding ability: models have seen the tasks, and many hard tasks have broken tests. OpenAI stopped reporting it and asked others to do the same.

Key claims

  • Contamination: every frontier model tested could reproduce the gold patch or verbatim problem details for some tasks.
  • Audit of 138 hard problems (ones o3 did not consistently solve over 64 runs), each reviewed by at least six engineers: 59.4% had flawed tests or descriptions that reject correct solutions.
  • Saturation: state of the art moved only from 74.9% to 80.9% over the preceding six months.
  • The 59.4% applies to the audited hard subset (about 27.6% of the dataset), not to all 500 tasks.
  • Recommendation: report SWE-bench Pro instead. Later third-party audits (unverified here) argue SWE-bench Pro has grading problems of its own; secondary reports that OpenAI later withdrew the endorsement have no primary source.

Feeds

Coding agents

Suggest changes →