Source: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020)

The MMLU paper: 57-subject multiple-choice benchmark of world knowledge and problem solving.

Contents
  • What: research paper, arXiv:2009.03300 (ICLR 2021)
  • Authors: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt
  • Date: 7 September 2020 (v3 12 January 2021)
  • Original: arxiv.org/abs/2009.03300

Summary

Introduces MMLU, a multiple-choice test across 57 subjects (elementary maths, US history, computer science, law and more) to measure knowledge acquired in pretraining, in zero- and few-shot settings.

Key claims

  • 57 tasks span STEM, humanities and social sciences at levels from elementary to professional.
  • The largest GPT-3 model beat random chance by almost 20 points on average; most models were near chance.
  • Models were lopsided across subjects and weak on socially important ones such as law and morality, and still far from expert level.

Feeds

Benchmarks

Suggest changes →