Source: Language Models Don't Always Say What They Think
Turpin et al. (2023): chain-of-thought explanations can rationalise answers driven by hidden biasing features without mentioning them.
Contents
- What: research paper (NeurIPS 2023), arXiv:2305.04388
- Authors: Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman
- Date: v1 2023-05-07, v2 2023-12-09
- Original: arXiv:2305.04388
Summary
Tests whether CoT explanations reflect the true reason for a model’s answer. By adding biasing features (for example, always making the correct option “(A)” in few-shot examples), the authors steer answers and show the written reasoning does not mention the bias and instead rationalises the biased answer.
Key claims
- CoT explanations can systematically misrepresent why a model answered as it did.
- Biasing features caused accuracy drops of up to 36% across 13 BIG-Bench Hard tasks (GPT-3.5, Claude 1.0).
- On a social-bias task, explanations justified stereotype-consistent answers without acknowledging the stereotype.
- Plausible-looking CoT is not evidence of faithful reasoning.