Source: Are We Done with MMLU? (Gema et al., 2024)

Paper estimating 6.49% of MMLU questions are wrong and releasing the re-annotated MMLU-Redux subset.

Contents
  • What: research paper, arXiv:2406.04127 (NAACL 2025)
  • Authors: Aryo Pradipta Gema, Joshua Ong Jun Leang, …, Pasquale Minervini (16 authors)
  • Date: 6 June 2024 (v3 10 January 2025)
  • Original: arxiv.org/abs/2406.04127

Summary

Manual re-annotation of MMLU finds label and question errors that blur what the benchmark measures.

Key claims

  • An estimated 6.49% of MMLU questions contain errors; 57% of the analysed Virology questions did.
  • MMLU-Redux: 5,700 re-annotated questions across all 57 subjects.
  • Re-evaluating on MMLU-Redux shows significant discrepancies with originally reported model scores.

Feeds

Benchmarks

Suggest changes →