Source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023)
The SWE-bench paper: 2,294 real GitHub issues from 12 Python repos, graded by running the repo's tests.
Contents
- What: research paper, arXiv:2310.06770 (ICLR 2024)
- Authors: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan
- Date: 10 October 2023 (v3 11 November 2024)
- Original: arxiv.org/abs/2310.06770
Summary
Each task gives a model a real repository snapshot and an issue description; the model must edit the code so that the issue is resolved, which is checked by executing tests from the pull request that originally fixed it.
Key claims
- 2,294 task instances drawn from 12 popular Python repositories.
- Tasks require coordinating changes across functions, classes and files and interacting with an execution environment.
- At release the best model, Claude 2, resolved 1.96% of issues.