Source: Large Language Models Can Accurately Predict Searcher Preferences (Thomas et al.)

Microsoft/Bing study showing GPT-4 relevance labels, with a prompt tuned against real searcher preferences, match or beat crowd labellers at a fraction of the cost.

Contents
  • What: research paper, arXiv:2309.10621, published at ACM SIGIR 2024
  • Authors: Paul Thomas, Seth Spielman, Nick Craswell, Bhaskar Mitra (Microsoft)
  • Date: v1 19 September 2023; v3 16 May 2024
  • Original: arxiv.org/abs/2309.10621

Summary

Relevance labels are the bottleneck of retrieval evaluation. The authors pick an LLM prompt by checking it against a small set of high-quality “gold” labels derived from real searcher feedback, then use it to label at scale. They report results from Bing production use and from TREC-Robust, plus a careful look at how sensitive the labels are to prompt wording.

Key claims

  • Labels use a 0-2 scale (2 highly relevant, 1 relevant, 0 not relevant). Prompt features tested: a rater role, the TREC description and narrative, asking for aspect scores (topicality, trustworthiness) before the overall score, and simulating multiple judges.
  • On TREC-Robust (binarised), Cohen’s kappa with the TREC assessors ranged from 0.20 to 0.64 across 32 prompt variants; the best prompt (description + narrative + aspects, no role, no multiple judges) reached 0.64. Asking for aspects helped most (+0.208); simulated multiple judges hurt (-0.127).
  • Paraphrasing the best prompt moved mean kappa between 0.50 and 0.72 with no clear pattern: small wording changes matter.
  • For comparison, the prior human-labeller agreement studies the paper cites report kappa of roughly 0.24 to 0.58.
  • At Bing, relative to a typical crowd worker, GPT-4 labels were 28% more accurate, about 10x the throughput, at about 1/20 of the cost.
  • System rankings from LLM labels agreed well with human labels (Kendall tau 0.77 to 0.86 across 110 TREC runs, depending on metric), but identifying the hardest queries was less consistent.
  • The LLM said “not relevant” more often than assessors (44% vs 33%); when it said relevant, it agreed with TREC 94% of the time.
  • Caveats: risk of optimising systems toward the LLM’s quirks, possible bias against long documents, and the need to keep human gold labels and oversight.

Feeds

Retrieval metrics

Suggest changes →