Source: MMLU-Pro (Wang et al., 2024)
Harder, reasoning-heavier MMLU successor with ten answer options and lower prompt sensitivity.
Contents
- What: research paper, arXiv:2406.01574 (NeurIPS 2024 Datasets and Benchmarks, spotlight)
- Authors: Yubo Wang, Xueguang Ma, Ge Zhang, …, Wenhu Chen (17 authors)
- Date: 3 June 2024 (v6 6 November 2024)
- Original: arxiv.org/abs/2406.01574
Summary
Responds to MMLU saturation by removing trivial and noisy questions, adding reasoning-focused ones, and expanding the options from four to ten.
Key claims
- Ten options instead of four lowers the payoff from guessing; accuracy drops 16-33 points versus MMLU.
- Across 24 prompt styles, score sensitivity falls from 4-5% on MMLU to about 2%.
- Chain-of-thought helps on MMLU-Pro, unlike on the original MMLU, indicating more reasoning content.