Source: GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Rein et al., 2023)
448 expert-written biology, physics and chemistry questions that skilled non-experts with web access mostly fail.
Contents
- What: research paper, arXiv:2311.12022 (COLM 2024)
- Authors: David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
- Date: 20 November 2023
- Original: arxiv.org/abs/2311.12022
Summary
Multiple-choice questions written and validated by domain PhDs, designed to be “Google-proof” so that they test expertise and reasoning rather than lookup, and intended for scalable-oversight research.
Key claims
- 448 questions in biology, physics and chemistry.
- PhD-level experts scored 65% (74% discounting mistakes they later acknowledged).
- Skilled non-experts scored 34% despite unrestricted web access and over 30 minutes per question.
- The best GPT-4 baseline scored 39% at release.
- The main set has 448 questions; the widely reported “Diamond” subset is the 198-question highest-agreement slice; the full “Extended” set has 546 (expert 64.8%, non-expert 34.1% there), per the paper body.