Source: Deliberative Alignment (Guan et al., OpenAI, 2024)

OpenAI's method for teaching reasoning models the text of a safety spec and training them to reason over it, used for o1 and o3-mini, lowering both jailbreak success and over-refusal.

Contents
  • What: research paper (arXiv:2412.16339)
  • Authors: Melody Y. Guan et al. (OpenAI; 15 authors including Boaz Barak and Amelia Glaese)
  • Date: 2024-12-20 (v2 2025-01-08)
  • Original: arxiv.org/abs/2412.16339

Summary

Instead of learning safe behaviour only from labelled examples, the model is taught the written safety policy and learns to cite and apply it in its chain of thought. Training data is generated by the model itself with the spec in context, filtered by a spec-aware judge, and then the spec is removed so the model must recall it.

Key claims

  • SFT data: a helpful-only reasoning model answers safety-category prompts with the relevant spec in its prompt, producing CoT that references the policy.
  • A judge model with the spec scores each completion several times; the minimum score is used for filtering.
  • SFT trains on (prompt without spec, CoT, answer), so policy recall becomes internal.
  • RL: the same judge gives the reward, but the CoT is hidden from it to avoid optimisation pressure that could make CoTs deceptive.
  • Used for OpenAI’s o-series models, including o1-preview, o1 and o3-mini (the SFT stage for all of them; the judge-based RL stage for o1 and o3-mini). Reported XSTest not-over-refuse: GPT-4o 0.88, o1 0.93; StrongREJECT jailbreak robustness: GPT-4o 0.37, o1 0.88.
  • Claims to move the Pareto frontier: fewer jailbreaks and fewer over-refusals at once, without human-written CoTs.

Feeds

LLM training pipeline

Suggest changes →