Source: Group Sequence Policy Optimization (GSPO)

Qwen team's 2025 RL algorithm that moves GRPO's importance ratio and clipping from the token level to the sequence level, stabilising MoE RL.

Contents
  • What: research paper (arXiv:2507.18071)
  • Author: Chujie Zheng, Shixuan Liu, et al., including An Yang and Junyang Lin (Qwen team)
  • Date: 2025-07-24 (rev. 2025-07-28)
  • Original: arXiv:2507.18071

Summary

GSPO argues that GRPO’s per-token importance ratios make training unstable. It computes the importance ratio from the likelihood of the whole sequence, and it clips and optimises at the sequence level.

Key claims

  • Importance ratio, clipping and optimisation all happen at the sequence level, not per token.
  • More stable and efficient than GRPO, and notably stabilises RL on mixture-of-experts models.
  • Credited with contributing to the gains in the latest Qwen3 models.

Feeds

Reinforcement fine-tuning

Suggest changes →