Source: Detecting hallucinations in large language models using semantic entropy (Nature, 2024)

Oxford paper that detects confabulations by sampling several answers, clustering them by meaning, and measuring entropy over meanings.

Contents
  • What: peer-reviewed paper, Nature 630, 625-630
  • Authors: Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, Yarin Gal (University of Oxford)
  • Date: 19 June 2024
  • Original: nature.com/articles/s41586-024-07421-0 (open access copy on PMC)

Summary

The paper targets one subtype of hallucination, confabulation: answers that are both wrong and arbitrary, changing with irrelevant factors such as the random seed. It samples several answers, groups them into meaning clusters using bidirectional entailment (A entails B and B entails A), and computes entropy over clusters rather than over token strings. High semantic entropy flags a likely confabulation, so the system can abstain or escalate.

Key claims

  • Confabulation is distinguished from consistent errors learned from bad training data, deliberate deception, and systematic reasoning failures; semantic entropy does not catch those.
  • Averaged over 30 task-model combinations, semantic entropy reached AUROC 0.790 versus 0.691 for naive (token) entropy, 0.698 for P(True) and 0.687 for embedding regression.
  • The method used ten generations per question, so it multiplies inference cost.
  • For long-form text it needs extra steps: decompose into claims, generate questions per claim, then apply the method.
  • Results depend on the entailment model; clustering can be over-sensitive (a full date versus a bare year counted as different meanings).

Feeds

Hallucination

Suggest changes →