Source: Large Language Models are Zero-Shot Reasoners

Kojima et al. (2022): appending "Let's think step by step" elicits chain-of-thought reasoning without any exemplars.

Contents
  • What: research paper (NeurIPS 2022), arXiv:2205.11916
  • Authors: Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa
  • Date: v1 2022-05-24, v4 2023-01-29
  • Original: arXiv:2205.11916

Summary

Shows that CoT does not require hand-written exemplars: a single trigger phrase placed before the answer makes large models reason step by step (“Zero-shot-CoT”). The method uses two calls: one to extract reasoning, one to extract a formatted answer.

Key claims

  • With text-davinci-002, Zero-shot-CoT raised MultiArith accuracy from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%; similar gains with PaLM 540B.
  • Two-stage prompting: (1) “Q: … A: Let’s think step by step.” produces a reasoning passage; (2) the prompt plus reasoning plus an answer trigger (e.g. “Therefore, among A through E, the answer is”) yields a parseable answer.
  • Gains grow sharply with model size; small models do not benefit.
  • Zero-shot-CoT generally underperforms carefully crafted few-shot CoT, showed no gains on commonsense tasks, and is sensitive to the trigger wording.

Feeds

Chain-of-thought

Suggest changes →