Source: LiveBench: A Challenging, Contamination-Limited LLM Benchmark (White et al., 2024)

Benchmark that refreshes questions monthly from recent sources and scores against objective ground truth without LLM judges.

Contents
  • What: research paper, arXiv:2406.19314 (ICLR 2025 spotlight)
  • Authors: Colin White, Samuel Dooley, Manley Roberts, …, Yann LeCun, Micah Goldblum (18 authors)
  • Date: 27 June 2024 (v2 18 April 2025)
  • Original: arxiv.org/abs/2406.19314

Summary

Limits contamination by time: questions come from recent math competitions, arXiv papers, news and datasets, and are refreshed monthly. Answers are scored automatically against ground truth, avoiding both LLM-judge bias and crowd preferences.

Key claims

  • Categories: math, coding, reasoning, language, instruction following, data analysis.
  • Includes harder, contamination-limited versions of Big-Bench Hard, AMPS and IFEval tasks.
  • Monthly question refresh; harder tasks added over time.
  • Objective scoring avoids judge errors on hard questions.

Feeds

Benchmarks

Suggest changes →