Source: Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min et al. (2022): in few-shot classification, label correctness matters little; label space, input distribution and format drive gains.
Contents
- What: research paper (EMNLP 2022), arXiv:2202.12837
- Authors: Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer
- Date: v1 2022-02-25, v2 2022-10-20
- Original: arXiv:2202.12837
Summary
Asks which parts of few-shot demonstrations actually help. Across 12 models including GPT-3, on classification and multiple-choice tasks, randomly replacing demonstration labels barely hurt accuracy. What mattered was exposure to the label space, the distribution of input text, and the overall format.
Key claims
- Ground-truth labels in demonstrations are largely not required for these tasks.
- Label space, input distribution and sequence format are the main drivers of in-context gains.
- Suggests demonstrations mostly tell the model what the task looks like rather than teaching the input-label mapping.