Source: MMLU-Pro (Wang et al., 2024)

Harder, reasoning-heavier MMLU successor with ten answer options and lower prompt sensitivity.

Contents
  • What: research paper, arXiv:2406.01574 (NeurIPS 2024 Datasets and Benchmarks, spotlight)
  • Authors: Yubo Wang, Xueguang Ma, Ge Zhang, …, Wenhu Chen (17 authors)
  • Date: 3 June 2024 (v6 6 November 2024)
  • Original: arxiv.org/abs/2406.01574

Summary

Responds to MMLU saturation by removing trivial and noisy questions, adding reasoning-focused ones, and expanding the options from four to ten.

Key claims

  • Ten options instead of four lowers the payoff from guessing; accuracy drops 16-33 points versus MMLU.
  • Across 24 prompt styles, score sensitivity falls from 4-5% on MMLU to about 2%.
  • Chain-of-thought helps on MMLU-Pro, unlike on the original MMLU, indicating more reasoning content.

Feeds

Benchmarks

Suggest changes →