Preference Optimization

Post-training that pushes a model toward preferred or rewarded outputs: RLHF with PPO, DPO and its variants, GRPO and RL with verifiable rewards, and RLAIF / Constitutional AI.

Contents

Preference optimization is the stage of fine-tuning that comes after SFT. Instead of imitating fixed demonstrations, the model learns from comparisons (A is better than B) or rewards (a score for each sampled output), and is pushed toward higher-preference behaviour while a KL constraint keeps it near a reference model. The family includes RLHF with a learned reward model and PPO (InstructGPT), offline methods such as DPO that skip the reward model and the RL loop (DPO), critic-free online RL such as GRPO with verifiable rewards (GRPO, R1), and variants that replace human labels with AI feedback (Constitutional AI, RLAIF).

All of them optimise roughly the same objective: maximise expected reward minus β·KL(π‖π_ref). The reference model π_ref is usually the SFT checkpoint (DPO).

flowchart TD
  P["Pre-trained base"] --> S["SFT on demonstrations"]
  S --> R1["RLHF: train reward model on human rankings"]
  R1 --> R2["PPO vs reward model + KL to SFT"]
  S --> D["DPO: offline loss on chosen/rejected pairs"]
  S --> G["GRPO: sample a group, score with rule/verifier, normalise within group"]
  H["Human preference labels"] --> R1
  H --> D
  AI["AI labeller + constitution (RLAIF)"] --> R1
  AI --> D
  V["Verifiable rewards: tests, exact answers, format"] --> G

Synthesis: the diagram combines the pipelines from the InstructGPT, DPO, CAI and DeepSeek papers.

RLHF with PPO (the InstructGPT recipe)

  1. SFT on labeller demonstrations (~13k prompts).
  2. Reward model (RM). Labellers ranked 4–9 responses per prompt (~33k prompts). A 6B RM was trained on all the pairwise comparisons; 175B RMs were unstable.
  3. PPO against the RM (~31k prompts), with a per-token KL penalty to the SFT model to limit reward over-optimisation.

Results: a 1.3B InstructGPT was preferred to 175B GPT-3. The costs: an “alignment tax” on some NLP benchmarks, reduced by mixing pre-training gradients (PPO-ptx). Inter-labeller agreement was only about 73–77%, so preference labels are noisy (InstructGPT). PPO needs four models in memory: policy, reference, reward model and a value model (critic) usually as large as the policy (GRPO).

DPO: Direct Preference Optimization

DPO shows that the KL-constrained RLHF optimum has a closed form. The policy itself defines an implicit reward r(x,y) = β·log(πθ(y|x)/π_ref(y|x)), so preferences can be fit with a logistic (Bradley–Terry) loss:

text
L = −log σ( β·[log πθ(y_w|x)/π_ref(y_w|x) − log πθ(y_l|x)/π_ref(y_l|x)] )
  • No reward model, no sampling during training, no RL loop. It is an offline classification loss (DPO).
  • The gradient weights each pair by how wrongly the implicit reward currently orders it. Without that weighting the model degenerates (DPO).
  • β sets how far the policy may move from the reference; higher β means less deviation. The paper used β=0.1 (0.5 for summarisation) and LR 1e-6 (DPO). TRL defaults to the same β=0.1 and LR 1e-6 (TRL).
  • Results: better sentiment control than PPO, and equal or better summarisation and dialogue quality (DPO).
  • TRL documents a practical caveat: DPO usually wins by pushing down the rejected completion rather than raising the chosen one. Monitor rewards/chosen, rewards/rejected, rewards/margins and rewards/accuracies (TRL).
  • Variants (TRL loss_type): ipo (identity transform, against overfitting), hinge (RSO/SLiC), robust and udpo (noisy labels), sigmoid_norm (SimPO-style length normalisation), apo_zero/apo_down (anchored), and more. Separate trainers cover KTO (unpaired good/bad labels), ORPO and CPO (no reference model). Online DPO samples fresh pairs during training (TRL).

GRPO and RL with verifiable rewards

GRPO (Group Relative Policy Optimization) is a PPO variant without a critic. For each prompt it samples a group of G outputs and scores them. Each output’s advantage is its reward normalised within the group, (r − mean)/std. The KL to the reference is added directly to the loss with an unbiased estimator. Dropping the value model saves substantial memory. DeepSeekMath used G=64 and KL coefficient 0.04 (GRPO). The paper also found an important limit: RL raised Maj@K but not Pass@K. It made already-reachable correct answers more likely rather than adding new capability (GRPO).

DeepSeek-R1 made this approach widely known:

  • R1-Zero ran GRPO straight on a base model with rule-based rewards (answer correctness via math checks or compiler tests, plus a format reward) and developed long chains of thought, self-verification and reflection.
  • Neural reward models were avoided for reasoning because they invite reward hacking at scale. Process reward models (PRMs) and MCTS were tried and abandoned.
  • R1-Zero’s outputs were hard to read and mixed languages. R1 fixed this with cold-start SFT, a language-consistency reward, rejection-sampling SFT and a final RL stage (R1).
  • See reasoning models and test-time compute.

TRL GRPOTrainer defaults (read 2026-11-14): num_generations=8, beta=0.0 (KL off, no reference model loaded), loss_type="dapo" (token-level normalisation; dr_grpo removes length bias), epsilon=0.2 clipping, scale_rewards="group", LR 1e-6. TRL notes that DeepSeek-R1 used β=0.001. Rewards are plain Python functions returning one float per completion, summed or weighted. Generation runs in vLLM, colocated or as a server (TRL). RL learns little per episode, so LoRA matches full fine-tuning here even at rank 1 (LoRA Without Regret). Unsloth suggests LR around 5e-6 for DPO/GRPO (Unsloth).

RLAIF and Constitutional AI

  • Constitutional AI (Anthropic). Human oversight is reduced to a short list of principles. In the supervised phase the model critiques and revises its own answers, then trains on the revisions. In the RL phase an AI picks the better of two samples, a preference model is trained on those choices, and RL runs against it (RLAIF). The result is harmless but non-evasive: it explains its objections instead of stonewalling (CAI).
  • RLAIF vs RLHF (Google). AI labels matched human labels on summarisation and on helpful and harmless dialogue. RLAIF helped even when the labeller was the same model as the policy. Direct RLAIF, which skips the RM and asks an LLM for rewards during RL, did better still (RLAIF). This is the same pattern as LLM-as-judge, applied to training.

Comparison

MethodDataOnline sampling?Models in memoryStrengthsWeaknesses
PPO-RLHFRankings → RMYesPolicy, ref, RM, criticGeneral preferences; matureComplex, unstable, expensive; RM hacking
DPO (and IPO, SimPO…)Chosen/rejected pairsNo (offline)Policy + ref (or precomputed ref log-probs)Simple, cheap, stableLimited to dataset coverage; can lower chosen likelihood; length bias
KTOUnpaired good/bad labelsNoPolicy + refUses thumbs-up/down logsWeaker signal than pairs
GRPO / RLVRPrompts + reward functionsYesPolicy (+ ref if β>0)No critic; great for verifiable tasksNeeds a reliable verifier; sharpens rather than expands ability
RLAIF / CAIAI labels from principlesEitherAs aboveScales labels cheaplyInherits the labeller’s biases

Sources: rows draw on InstructGPT, DPO, TRL, GRPO, CAI and RLAIF. The “weaknesses” column for KTO and RLAIF is Synthesis.

Managed options (as of 2026-11-14)

  • OpenAI: DPO on gpt-4.1 and gpt-4.1-mini, for summary focus and chat tone. The platform is winding down: existing customers can’t create jobs from 2027-01-06, and both base models shut down 2027-01-14 (OpenAI, deprecations).
  • Vertex AI: preference tuning on Gemini 2.5 Flash/Flash-Lite with hundreds to thousands of pairs, and RL fine-tuning with reward functions on Gemini 3.5 Flash and 3.1 Flash-Lite with dozens to a few hundred prompts (Vertex AI).
  • Bedrock: reinforcement fine-tuning with AWS Lambda reward functions, using uploaded prompts or invocation logs (Bedrock).

Failure modes

  • Reward hacking and over-optimisation. The policy exploits a flawed RM or verifier, which is why KL penalties exist and why R1 avoided neural RMs (InstructGPT, R1).
  • Alignment tax. Regressions on general benchmarks (InstructGPT).
  • Noisy preferences. Labellers disagree about a quarter of the time (InstructGPT). Robust losses help (TRL).
  • Length and verbosity bias. Handled by normalised losses such as sigmoid_norm and dr_grpo (TRL).
  • Readability and language drift under pure RL (R1).

When to use which

Synthesis:

  • Start with SFT.
  • Add DPO when you can collect “A beats B” judgements (human, or LLM-as-judge) for subjective qualities such as tone, helpfulness or safety style.
  • Use GRPO/RFT when correctness can be checked by a program: tests, exact answers, schema validity (structured outputs), tool-call success.
  • Reserve PPO-RLHF for labs with large-scale infrastructure.
  • For small models, distilling from a stronger reasoner beat running RL directly in R1’s experiments (R1).

Sources

Suggest changes →