Source: The illusion of a perfect metric (Oliva et al., 2025)

Survey arguing no automatic text metric, including LLM-as-judge and RAG metrics, is reliably validated against human judgment.

Contents
  • What: research paper, arXiv:2508.13816 (RANLP 2025)
  • Authors: Maria Paz Oliva, Adriana Correia, Ivan Vankov, Viktor Botev
  • Date: 2025-08-19
  • Original: arxiv.org/abs/2508.13816

Summary

Reviews the evolution from lexical to semantic to LLM-based evaluation metrics and finds the same validation problems at every stage.

Key claims

  • No single automatic metric has become definitive; each captures only some aspects of quality.
  • Metric effectiveness varies by task and dataset; correlations with human judgment are inconsistent and validation practice is unstructured.
  • These problems persist for LLM-as-a-judge and for RAG evaluation.
  • Recommendation: choose metrics for the task, combine complementary evaluations, and invest in better validation.

Feeds

RAGAS and DeepEval

Suggest changes →