Source: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020)
The MMLU paper: 57-subject multiple-choice benchmark of world knowledge and problem solving.
Contents
- What: research paper, arXiv:2009.03300 (ICLR 2021)
- Authors: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt
- Date: 7 September 2020 (v3 12 January 2021)
- Original: arxiv.org/abs/2009.03300
Summary
Introduces MMLU, a multiple-choice test across 57 subjects (elementary maths, US history, computer science, law and more) to measure knowledge acquired in pretraining, in zero- and few-shot settings.
Key claims
- 57 tasks span STEM, humanities and social sciences at levels from elementary to professional.
- The largest GPT-3 model beat random chance by almost 20 points on average; most models were near chance.
- Models were lopsided across subjects and weak on socially important ones such as law and morality, and still far from expert level.