Source: Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning (Yuan et al., 2025)
Paper showing that batch size, GPU count and GPU type change greedy-decoding results under BF16, with up to 9% accuracy spread on reasoning models; FP32 nearly eliminates it and LayerCast is proposed as a cheaper middle ground.
Contents
- What: research paper (arXiv 2506.09501)
- Author: Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, Zirui Liu (Rice University, University of Minnesota, Adobe)
- Date: v1 11 June 2025
- Original: arxiv.org/abs/2506.09501
Summary
Measures how much “deterministic” greedy decoding changes when only the serving configuration changes, across 12 configurations (2 GPU types: L40S and A100; 2 GPU counts; batch sizes 8/16/32). Attributes the spread to non-associative floating-point arithmetic at limited precision, and shows numerical precision is the main lever.
Key claims
- Models: DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct; benchmarks AIME'24, MATH500, LiveCodeBench.
- Under BF16 greedy decoding, accuracy standard deviation on AIME'24 reached 9.15% (R1-Distill-Qwen-7B); FP16 5.74%; FP32 0.
- Response length varied by up to roughly 9,000 tokens (std) under BF16 on AIME'24 for the same model; FP32 gave 0 there.
- The reasoning (R1-distilled) models, which emit long outputs, produced the headline spreads (AIME'24 accuracy and length).
- LayerCast: keep weights in BF16, upcast to FP32 just before each matmul, compute in FP32; about 34% less memory than full FP32, released as a vLLM patch.
- Recommendations: with compute to spare, sample at non-zero temperature over multiple runs and report error bars; for single-run greedy evaluation, use FP32.