Source: Prompting Science Report 2: The Decreasing Value of Chain of Thought
Meincke, Mollick, Mollick and Shapiro (2025): explicit CoT prompts give small, variable gains on non-reasoning models and only marginal gains on reasoning models at much higher time and token cost.
Contents
- What: technical report (Wharton Generative AI Labs), arXiv:2506.07142
- Authors: Lennart Meincke, Ethan Mollick, Lilach Mollick, Dan Shapiro
- Date: 2025-06-08
- Original: arXiv:2506.07142
Summary
Re-measures the benefit of “think step by step” style prompting on 2025-era models, using GPQA Diamond (198 PhD-level multiple-choice questions) with 25 runs per question per condition.
Key claims
- Non-reasoning models: CoT prompting gives a small average improvement but adds answer variability, causing occasional errors on questions the model would otherwise get right.
- Many recent non-reasoning models already reason step by step unprompted, shrinking the gain.
- Reasoning models: explicit CoT gives marginal accuracy gains while substantially increasing time and tokens.