Source: Are We Done with MMLU? (Gema et al., 2024)
Paper estimating 6.49% of MMLU questions are wrong and releasing the re-annotated MMLU-Redux subset.
Contents
- What: research paper, arXiv:2406.04127 (NAACL 2025)
- Authors: Aryo Pradipta Gema, Joshua Ong Jun Leang, …, Pasquale Minervini (16 authors)
- Date: 6 June 2024 (v3 10 January 2025)
- Original: arxiv.org/abs/2406.04127
Summary
Manual re-annotation of MMLU finds label and question errors that blur what the benchmark measures.
Key claims
- An estimated 6.49% of MMLU questions contain errors; 57% of the analysed Virology questions did.
- MMLU-Redux: 5,700 re-annotated questions across all 57 subjects.
- Re-evaluating on MMLU-Redux shows significant discrepancies with originally reported model scores.