Reasoning Models
LLMs trained with reinforcement learning to produce a long hidden chain of thought before answering, trading tokens and latency for accuracy on verifiable, multi-step problems.
Contents
A reasoning model (also “large reasoning model”, LRM) is an LLM post-trained, mainly with reinforcement learning on tasks whose answers can be checked automatically, to generate a long internal chain of thought before its final answer. In that reasoning it plans, tries approaches, checks itself and backtracks. OpenAI’s o1 (September 2024) introduced the category and showed that accuracy improves with both RL training compute and inference-time thinking. DeepSeek-R1 (January 2025) published an open recipe. As of November 2026, every frontier family ships reasoning as a mode or a default: OpenAI’s GPT-5.x/6 with reasoning.effort, Claude’s adaptive/extended thinking, Gemini’s thinking_level, DeepSeek’s thinking mode (OpenAI, DeepSeek-R1, OpenAI guide, Claude docs, Gemini docs, DeepSeek docs).
The difference from chain-of-thought prompting is where the reasoning comes from. CoT asks a general model to show its work. A reasoning model has been trained to reason, and the length and quality of that reasoning are learned behaviours. This is why it can be scaled with a budget or effort knob (test-time compute).
How they are trained
The core ingredient is RL with verifiable rewards (RLVR). Sample completions for problems with a checkable answer, such as maths with a known result, code with unit tests, or instructions with programmatic constraints. Reward the completions that pass and update the policy. Tulu 3 named the technique: a constant reward on success, in place of a learned reward model (Tulu 3).
DeepSeek-R1 is the best-documented instance (DeepSeek-R1):
- R1-Zero applies RL directly to a base model, with no SFT. It uses GRPO, which samples a group of answers per prompt and uses the group’s mean reward as the baseline, so no critic network is needed. Rewards are purely rule-based: answer accuracy plus a format reward for wrapping reasoning in think tags. Neural reward models were rejected because the policy learns to hack them.
- With nothing telling it to, the model’s responses grow longer over training, and reflection and backtracking appear on their own. AIME 2024 pass@1 rose from 15.6% to 71.0%.
- R1-Zero’s output was hard to read and mixed languages. R1 therefore uses four stages: cold-start SFT on a few thousand clean long-CoT examples → reasoning RL (plus a language-consistency reward) → rejection-sample about 800K examples and SFT again → RL across all scenarios, with rule rewards for reasoning and reward models for helpfulness and harmlessness.
- What did not work: process reward models (steps are hard to define and easy to hack) and MCTS (the token search space is too large).
- Distillation: fine-tuning small dense models on R1’s 800K traces beat running RL on those small models directly. Qwen-32B scored 72.6% on AIME distilled against 47.0% with RL. See knowledge distillation.
flowchart LR B["Base model"] --> CS["Cold-start SFT<br/>(few thousand long CoTs)"] CS --> RL1["Reasoning RL (GRPO)<br/>rule rewards: correct + format + language"] RL1 --> RS["Rejection sampling<br/>~800K SFT samples"] RS --> SFT2["SFT on base"] SFT2 --> RL2["All-scenario RL<br/>rules + reward models"] RL2 --> R1["Reasoning model"] RS -. "distil" .-> SM["Small dense models"]
Diagram: DeepSeek-R1 pipeline as described in the paper.
OpenAI described o1’s training only at a high level: a data-efficient, large-scale RL algorithm that teaches the model to use its chain of thought productively (OpenAI). Synthesis: verifiable rewards explain why reasoning models gained most in maths, code and STEM, where a cheap automatic checker exists. They gained less in open-ended writing, where reward has to come from preference models (see preference optimization).
How they are used: API surface (Nov 2026)
| Vendor | Depth control | Reasoning visible? | Multi-turn / tools rule |
|---|---|---|---|
| OpenAI (Responses API) | reasoning.effort: none…max (model-dependent; default medium); reasoning.mode standard/pro | No. Optional summaries (reasoning.summary) | Pass reasoning items back with tool outputs; encrypted_content when store: false |
| Anthropic (Messages API) | thinking: {type: "adaptive"} + output_config.effort low…max (default high; medium on Opus 5.5 / Haiku 5.5); legacy budget_tokens | Summaries only (display: "summarized"), or omitted | Pass thinking blocks back unmodified within a tool-use turn |
| Google Gemini | thinking_level minimal…high (xhigh on one model) | Summaries opt-in | Resend thought blocks/signatures unmodified when stateless |
| DeepSeek | thinking.type + reasoning_effort low/high/max | Yes, full reasoning_content | With tools, pass back all prior reasoning_content or get a 400 |
Sources: OpenAI guide, Claude thinking, Claude steering, Gemini, DeepSeek.
Common mechanics across vendors:
- Reasoning tokens are billed as output tokens even when hidden, and they count toward the output cap (
max_tokens/max_output_tokens). OpenAI says usage runs from hundreds to tens of thousands of tokens per request and advises reserving at least 25K at first. Hitting the cap can return no visible answer at all (OpenAI guide, Gemini). - Sampling knobs are restricted. Recent Claude models reject non-default
temperature/top_p/top_k(and from API version2026-10-21every Claude model rejects them), and DeepSeek ignores them in thinking mode (Claude thinking, DeepSeek). See sampling parameters. - Opaque continuity tokens (signatures, encrypted reasoning) must be round-tripped unmodified across function calls. Dropping them is the most common integration bug (Synthesis, from all four vendors’ rules).
Prompting differences
- Don’t add “think step by step”. The model already reasons, and the instruction can hurt (OpenAI best practices). Explicit CoT prompts give reasoning models marginal gains at a large time and token cost (Meincke et al.).
- State the goal, constraints, output format and what “done” means, rather than prescribing the steps (OpenAI guide). Start zero-shot. R1 in particular degrades with few-shot examples (DeepSeek-R1).
- Use the effort control before prompt wording to change how much the model thinks. It is calibrated, while prompt steering depends on wording and doesn’t work on some models (Claude steering).
When they help, and when not
OpenAI’s framing is reasoning models as planners and GPT-style models as workhorses: a reasoning model plans and makes the decisions, and a fast model executes the steps (OpenAI best practices).
Use them for: maths, code and debugging across files, multi-step agentic planning, ambiguous instructions, finding relationships across many dense documents, and grading other models’ outputs (OpenAI best practices). See also LLM-as-judge and coding agents. Gemini’s guidance is the same in other terms: minimal thinking for retrieval and classification, maximum for hard coding, maths and planning (Gemini).
Avoid or turn down for: latency-sensitive chat, classification, extraction, simple retrieval, and high-volume pipelines where a non-reasoning call already passes your evals. The specific risks:
- Overthinking. o1-style models spend many tokens on trivial problems for no accuracy gain (Chen et al.).
- Inverse scaling. On some tasks longer reasoning lowers accuracy: distraction by irrelevant details, overfitting to familiar framings, drifting into spurious correlations, losing track in long deductions (Gema et al.).
- Weaker general skills. R1 trailed its non-reasoning sibling on function calling, multi-turn dialogue and JSON output (DeepSeek-R1). Synthesis: test structured outputs and tool use on the reasoning variant specifically, rather than assuming parity.
- Unfaithful reasoning. Visible reasoning often omits what actually drove the answer. Claude 3.7 Sonnet mentioned a hint it used about 25% of the time, R1 about 39%. Treat thinking as a debugging aid, not an audit trail (Anthropic).
- Cost and latency. Tokens at output price, plus time to first visible token. Hidden reasoning also makes streaming UX harder (see cost and latency optimization).
Synthesis, as a practical decision rule: build the eval first (LLM evaluation). Run the task at no or low reasoning, then step effort up until quality plateaus, and plot quality against cost and p95 latency. Route by difficulty: cheap model or low effort by default, escalating hard cases. This mirrors Snell et al.’s finding that the best use of test-time compute depends on how hard the question is (Snell et al.). See model selection.
Sources
- Learning to Reason with LLMs (OpenAI)
- DeepSeek-R1
- Tulu 3: RLVR
- Scaling LLM Test-Time Compute Optimally
- OpenAI reasoning models guide
- OpenAI reasoning best practices
- Claude API docs: Thinking
- Claude API docs: Steering thinking
- Gemini API docs: Thinking
- DeepSeek API docs: Thinking mode
- Inverse Scaling in Test-Time Compute
- Do NOT Think That Much for 2+3=?
- Reasoning models don’t always say what they think
- The Decreasing Value of Chain of Thought