Source: Hugging Face TRL Documentation
Documentation for TRL, Hugging Face's post-training library (SFT, DPO, GRPO, reward modelling, distillation).
Contents
- What: library documentation
- Author: Hugging Face
- Date: v1.x (TRL v1 announced 2026-03-31; read 2026-11-14)
- Original: TRL docs
Summary
TRL provides trainers for the main post-training methods on top of transformers, with PEFT, vLLM, DeepSpeed and Unsloth integrations. Read here: the index, SFTTrainer, DPOTrainer and GRPOTrainer pages.
Key claims
- Stable trainers: SFT, DPO, KTO, Reward, GRPO, AsyncGRPO, RLOO, Distillation; experimental: ORPO, CPO, Online DPO, GKD and others.
- SFTTrainer accepts language-modelling or prompt-completion data, standard or conversational, and applies chat templates; loss defaults to completion-only for prompt-completion data;
assistant_only_lossandpackingare options. Default LR 2e-5 (≈1e-4 suggested for adapters), max_length 1024. For base models with a built-in chat template (e.g. Qwen2.5-1.5B) seteos_token="<|im_end|>". - DPOTrainer takes prompt/chosen/rejected data; default β = 0.1, LR 1e-6 (≈1e-5 suggested for adapters), loss
sigmoid, with alternatives such as ipo, hinge, robust and sigmoid_norm (SimPO-style). With no ref_model, the initial policy is the reference. - DPO logs rewards/chosen, rewards/rejected, rewards/margins and rewards/accuracies (implicit rewards).
- GRPOTrainer takes Python reward functions returning one float per completion; defaults num_generations 8, β = 0.0 (no KL, no reference model loaded; the docs note DeepSeek-R1 used 0.001), loss_type
dapo, epsilon 0.2, scale_rewardsgroup, LR 1e-6; vLLM colocate or server mode for generation.
Feeds
Fine-tuning · Preference optimization · PEFT and LoRA · Knowledge distillation