Source: The Curious Case of Neural Text Degeneration (Holtzman et al., 2020)

Introduces nucleus (top-p) sampling and shows that likelihood-maximising decoding produces bland, repetitive text.

Contents
  • What: research paper (arXiv 1904.09751, ICLR 2020)
  • Author: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi
  • Date: 2019-04-22 (v1); revised 2020-02-14 (v2)
  • Original: arXiv:1904.09751

Summary

Studies why decoding a strong language model by maximising likelihood (greedy or beam search) yields degenerate text, and proposes sampling only from the smallest set of tokens whose cumulative probability exceeds p, a set whose size adapts to the shape of the distribution.

Key claims

  • Maximisation-based decoding yields bland and strangely repetitive text even from good models.
  • Human text has different distributional properties from machine text; humans do not always pick the most likely word.
  • Decoding strategy alone can change quality dramatically for the same model.
  • Nucleus sampling truncates the unreliable low-probability tail with a dynamic cut-off, preserving diversity without hurting fluency or coherence.

Feeds

Sampling parameters

Suggest changes →