Source: GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Rein et al., 2023)

448 expert-written biology, physics and chemistry questions that skilled non-experts with web access mostly fail.

Contents
  • What: research paper, arXiv:2311.12022 (COLM 2024)
  • Authors: David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman
  • Date: 20 November 2023
  • Original: arxiv.org/abs/2311.12022

Summary

Multiple-choice questions written and validated by domain PhDs, designed to be “Google-proof” so that they test expertise and reasoning rather than lookup, and intended for scalable-oversight research.

Key claims

  • 448 questions in biology, physics and chemistry.
  • PhD-level experts scored 65% (74% discounting mistakes they later acknowledged).
  • Skilled non-experts scored 34% despite unrestricted web access and over 30 minutes per question.
  • The best GPT-4 baseline scored 39% at release.
  • The main set has 448 questions; the widely reported “Diamond” subset is the 198-question highest-agreement slice; the full “Extended” set has 546 (expert 64.8%, non-expert 34.1% there), per the paper body.

Feeds

Benchmarks

Suggest changes →