Source: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)

Paper describing Chatbot Arena's crowdsourced, anonymous pairwise battles and statistical ranking of LLMs.

Contents
  • What: research paper, arXiv:2403.04132 (ICML 2024)
  • Authors: Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, Ion Stoica
  • Date: 7 March 2024
  • Original: arxiv.org/abs/2403.04132

Summary

Chatbot Arena ranks models from crowdsourced pairwise comparisons: users chat with two anonymous models side by side and vote for the better answer, and the votes are fitted to a statistical ranking.

Key claims

  • Pairwise human preference on live, user-written prompts avoids a fixed test set.
  • More than 240K votes had been collected at the time of writing.
  • Crowdsourced votes agree well with expert raters.

Feeds

Benchmarks · LLM-as-judge

Suggest changes →