Source: LiveBench: A Challenging, Contamination-Limited LLM Benchmark (White et al., 2024)
Benchmark that refreshes questions monthly from recent sources and scores against objective ground truth without LLM judges.
Contents
- What: research paper, arXiv:2406.19314 (ICLR 2025 spotlight)
- Authors: Colin White, Samuel Dooley, Manley Roberts, …, Yann LeCun, Micah Goldblum (18 authors)
- Date: 27 June 2024 (v2 18 April 2025)
- Original: arxiv.org/abs/2406.19314
Summary
Limits contamination by time: questions come from recent math competitions, arXiv papers, news and datasets, and are refreshed monthly. Answers are scored automatically against ground truth, avoiding both LLM-judge bias and crowd preferences.
Key claims
- Categories: math, coding, reasoning, language, instruction following, data analysis.
- Includes harder, contamination-limited versions of Big-Bench Hard, AMPS and IFEval tasks.
- Monthly question refresh; harder tasks added over time.
- Objective scoring avoids judge errors on hard questions.