Source: Cumulated Gain-Based Evaluation of IR Techniques (Järvelin and Kekäläinen)
The 2002 ACM TOIS paper that introduced cumulated gain, DCG and normalized DCG for graded relevance judgments.
Contents
- What: journal article, ACM Transactions on Information Systems 20(4), pp. 422-446, DOI 10.1145/582415.582418
- Authors: Kalervo Järvelin, Jaana Kekäläinen (University of Tampere)
- Date: October 2002
- Original: ACM DL · author-hosted PDF
Summary
Argues that binary precision and recall hide the difference between retrieving highly relevant and marginally relevant documents, and proposes three measures for graded relevance: cumulated gain (CG), discounted cumulated gain (DCG) and their normalized versions relative to an ideal ranking (nCG, nDCG). Tested on TREC-style data re-judged on a four-point scale (0 to 3).
Key claims
- CG at rank i is the running sum of the graded relevance scores of the documents at ranks 1..i.
- DCG divides each document’s gain by log_b of its rank, so late documents count less. The base b models user patience (a larger base means a gentler discount). In the original definition ranks below b are not discounted at all, because log 1 = 0 and a discount there would boost the score.
- Normalized (D)CG divides the vector component-wise by the ideal (D)CG vector, giving 1 for ideal performance at every rank and values in [0, 1) otherwise.
- The ideal vector is built from the topic’s whole recall base (all judged relevant documents, sorted by grade), not from the documents the system happened to retrieve.
- The measures reward systems that put highly relevant documents first, and the curves are easier to interpret from a user’s point of view than precision-recall curves.