Source: The illusion of a perfect metric (Oliva et al., 2025)
Survey arguing no automatic text metric, including LLM-as-judge and RAG metrics, is reliably validated against human judgment.
Contents
- What: research paper, arXiv:2508.13816 (RANLP 2025)
- Authors: Maria Paz Oliva, Adriana Correia, Ivan Vankov, Viktor Botev
- Date: 2025-08-19
- Original: arxiv.org/abs/2508.13816
Summary
Reviews the evolution from lexical to semantic to LLM-based evaluation metrics and finds the same validation problems at every stage.
Key claims
- No single automatic metric has become definitive; each captures only some aspects of quality.
- Metric effectiveness varies by task and dataset; correlations with human judgment are inconsistent and validation practice is unstructured.
- These problems persist for LLM-as-a-judge and for RAG evaluation.
- Recommendation: choose metrics for the task, combine complementary evaluations, and invest in better validation.