Source: Tulu 3 (Lambert et al., 2024) — RLVR

Ai2's open post-training recipe that named Reinforcement Learning with Verifiable Rewards: RL against a correctness check instead of a learned reward model.

Contents
  • What: research paper / open post-training recipe (arXiv 2411.15124)
  • Authors: Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. (Allen Institute for AI)
  • Date: 2024-11-22 (v5 2025-04-14)
  • Original: arXiv:2411.15124

Summary

Tulu 3 is a fully open post-training pipeline: SFT, then DPO, then a final stage the authors call Reinforcement Learning with Verifiable Rewards (RLVR). RLVR replaces the learned reward model of RLHF with a programmatic check of whether the completion is correct.

Key claims

  • RLVR gives a constant reward when a completion passes a verifiable check, and nothing otherwise. It trains on these verifiable rewards rather than on a reward model.
  • Target domains: grade-school and competition math (GSM8K, MATH) and precise instruction following with checkable constraints (IFEval). Datasets for each are released.
  • RLVR improved GSM8K, MATH and IFEval scores.

Feeds

Reasoning models

Suggest changes →