Source: Sun et al. 2023: RankGPT (Is ChatGPT Good at Search?)
EMNLP 2023 paper showing instructed LLMs can rerank as well as supervised rerankers, and distilling that into small models.
Contents
- What: research paper (EMNLP 2023), arXiv:2304.09542
- Author: Weiwei Sun, Lingyong Yan, Xinyu Ma, et al.
- Date: 2023-04-19 (v3 2024-12-28)
- Original: arXiv:2304.09542
Summary
Studies generative LLMs (ChatGPT, GPT-4) as rerankers that output a permutation of candidate passages, and introduces a contamination-free test set.
Key claims
- Properly instructed LLMs are competitive with, and sometimes better than, supervised rerankers on standard IR benchmarks.
- NovelEval, a test set built on recent knowledge, addresses training-data contamination.
- Permutation distillation: a distilled 440M model beat a 3B supervised model on BEIR.