Source: Es et al. 2023: RAGAS, Automated Evaluation of Retrieval Augmented Generation

Paper introducing Ragas: reference-free, LLM-computed faithfulness, answer relevance and context relevance metrics.

Contents
  • What: research paper (EACL 2024 demo), arXiv:2309.15217
  • Author: Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert
  • Date: 2023-09-26 (v2 2025-04-28)
  • Original: arXiv:2309.15217

Summary

Proposes metrics that score a RAG pipeline without human ground-truth answers, by asking an LLM to decompose and check outputs.

Key claims

  • Faithfulness: split the answer into statements, check each against the retrieved context; score = supported / total.
  • Answer relevance: generate n questions from the answer, embed them, average cosine similarity to the original question.
  • Context relevance: extract the sentences of the context needed to answer; score = extracted / total sentences.
  • Agreement with human judgments on WikiEval: 0.95 faithfulness, 0.78 answer relevance, 0.70 context relevance, beating direct GPT scoring baselines.
  • Context relevance was the hardest dimension to evaluate.

Feeds

Retrieval-augmented generation

Suggest changes →