Source: Let Me Speak Freely? (Format Restrictions Study)

Tam et al. (2024): format restrictions such as JSON mode can measurably degrade LLM reasoning, more so the stricter the constraint.

Contents
  • What: research paper (arXiv:2408.02442)
  • Author/publisher: Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen
  • Date: August 2024 (v3 Oct 2024)
  • Original: arXiv:2408.02442

Summary

Empirical study of whether forcing outputs into formats like JSON or XML changes what LLMs can do. Compares free-form generation with format-restricted variants on reasoning and knowledge tasks.

Key claims

  • Format restrictions produce a significant drop in reasoning ability on the tested tasks.
  • Stricter constraints tend to cause larger degradation.
  • The effect is task-dependent: reasoning suffers most.

Feeds

Structured outputs

Suggest changes →