Source: DAPO (Yu et al., 2025)

ByteDance Seed / Tsinghua paper giving an open, large-scale RL recipe (DAPO) that fixes four failure modes of vanilla GRPO on long chain-of-thought training.

Contents
  • What: research paper with an open-source system (arXiv:2503.14476), built on verl
  • Author: Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. (35 authors, ByteDance Seed and collaborators)
  • Date: 2025-03-18 (v2 2025-05-20)
  • Original: arXiv:2503.14476

Summary

DAPO stands for Decoupled Clip and Dynamic sAmpling Policy Optimization. The authors’ baseline GRPO run on Qwen2.5-32B base reached only 30 on AIME 2024. They trace the gap to four problems and fix each one. The resulting recipe reaches 50, ahead of DeepSeek’s R1-Zero-Qwen-32B (47), in half the training steps. Code and data are released.

Key claims

  • Clip-Higher: separate clip bounds, ε_low = 0.2 and ε_high = 0.28. Vanilla clipping caps how fast low-probability tokens can rise, so policy entropy collapses quickly. A looser upper bound keeps exploration alive.
  • Dynamic sampling: a prompt whose sampled answers are all correct, or all wrong, has zero group advantage and gives no gradient. DAPO over-samples and keeps only prompts with mixed accuracy until the batch is full.
  • Token-level policy-gradient loss: GRPO averages inside each sequence and then across sequences, so each token in a long answer gets less weight. DAPO averages over all tokens in the group. This lets it penalise gibberish and repetition in long outputs.
  • Overlong reward shaping: truncated answers are masked out of the loss instead of punished. A soft length penalty applies in a 4,096-token window before the 20,480-token cap.
  • The KL penalty is dropped, because long-CoT RL is expected to move far from the initial model.
  • The reward is rule-based: +1 if the answer is equivalent to the reference, −1 otherwise. Answers were converted to integers so they parse reliably.

Feeds

Reinforcement fine-tuning

Suggest changes →