Source: The TREC-8 Question Answering Track Report (Voorhees)
Report on the first large-scale open-domain QA evaluation, whose ranked-answer scoring popularised mean reciprocal rank (MRR).
Contents
- What: track overview report, Proceedings of the 8th Text REtrieval Conference (TREC-8), pp. 77-82
- Author: Ellen M. Voorhees (NIST)
- Date: conference held 1999; NIST lists the publication as 11 December 2000
- Original: NIST publication page · PDF
Summary
TREC-8 ran the first large-scale evaluation of question answering systems not restricted to a domain. Systems returned a short ranked list of answer strings per question, with 50-byte and 250-byte answer-length limits. The scoring rule, now called mean reciprocal rank, became a standard metric for “one right answer” retrieval tasks.
Key claims
- Each run submitted up to five ranked responses per question; a question scores 1/rank of the first correct response, or 0 if none of the five is correct; the run score is the mean over questions (MRR). (The scoring details are from the report as described by secondary references, e.g. Wikipedia: Mean reciprocal rank; the NIST PDF is a scanned image.)
- The most accurate systems found a correct response for more than two thirds of the questions.
- Simple bag-of-words approaches worked well for 250-byte answers; 50-byte answers needed more sophisticated processing.