Source: Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang et al. (2022): sample many chain-of-thought paths and take a majority vote over final answers instead of greedy decoding.
Contents
- What: research paper (ICLR 2023), arXiv:2203.11171
- Authors: Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou
- Date: v1 2022-03-21, v4 2023-03-07
- Original: arXiv:2203.11171
Summary
Replaces greedy decoding of a single chain of thought with sampling several diverse reasoning paths and marginalising over them by picking the most frequent final answer. The intuition: a hard problem admits multiple valid reasoning routes that converge on the same correct answer, while errors scatter.
Key claims
- Absolute gains over CoT: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4%, ARC-challenge +3.9%.
- Main experiments sample 40 outputs per run; the authors note performance saturates quickly and suggest starting with 5 or 10 paths “to realize most of the gains”.
- Requires no extra training, verifier or fine-tuning; only sampling at non-zero temperature plus an answer-aggregation step.
- Works when answers come from a fixed set or can be compared exactly (numbers, options).