Source: Let Me Speak Freely? (Format Restrictions Study)
Tam et al. (2024): format restrictions such as JSON mode can measurably degrade LLM reasoning, more so the stricter the constraint.
Contents
- What: research paper (arXiv:2408.02442)
- Author/publisher: Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen
- Date: August 2024 (v3 Oct 2024)
- Original: arXiv:2408.02442
Summary
Empirical study of whether forcing outputs into formats like JSON or XML changes what LLMs can do. Compares free-form generation with format-restricted variants on reasoning and knowledge tasks.
Key claims
- Format restrictions produce a significant drop in reasoning ability on the tested tasks.
- Stricter constraints tend to cause larger degradation.
- The effect is task-dependent: reasoning suffers most.