Source: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)
Paper describing Chatbot Arena's crowdsourced, anonymous pairwise battles and statistical ranking of LLMs.
Contents
- What: research paper, arXiv:2403.04132 (ICML 2024)
- Authors: Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, Ion Stoica
- Date: 7 March 2024
- Original: arxiv.org/abs/2403.04132
Summary
Chatbot Arena ranks models from crowdsourced pairwise comparisons: users chat with two anonymous models side by side and vote for the better answer, and the votes are fitted to a statistical ranking.
Key claims
- Pairwise human preference on live, user-written prompts avoids a fixed test set.
- More than 240K votes had been collected at the time of writing.
- Crowdsourced votes agree well with expert raters.