OpenAI Responses API

OpenAI's primary generation endpoint: typed input/output items, optional server-side state, hosted tools, encrypted reasoning, compaction and background mode, and how it differs from Chat Completions.

Contents

The Responses API (POST /v1/responses) is OpenAI’s main endpoint for model calls, launched in March 2025 to merge Chat Completions’ simplicity with the Assistants API’s tools and state (InfoQ). A request takes instructions plus an input list of typed items; the response is an output list of items (message, reasoning, function_call, hosted-tool calls) rather than a single chat message. Unlike Chat Completions it can store conversation state on OpenAI’s side, run hosted tools inside the request, and carry encrypted reasoning between turns (migration guide). As of November 2026 it is the only route to the full GPT-6 feature set: GPT-6 Astra and GPT-6.1 Sol support tool calling only through Responses, and from GPT-5.4 on Chat Completions allows tool calling only with reasoning_effort: none (reasoning guide, migration guide), and the Assistants API was sunset on 2026-08-26 (migration guide).

Request and response shape

AspectChat CompletionsResponses
Unitmessages in and choices[].message outtyped items in input and output
System promptsystem/developer messagetop-level instructions
Candidatesn up to 128one generation, no n
Function toolnested under function (externally tagged)fields at top level (internally tagged)
Strict schemasoff unless strict: truetried when strict is omitted, falls back to non-strict if the schema is incompatible
Structured outputresponse_formattext.format
Hosted toolsnone (except search models)web search, file search, code interpreter, shell, computer use, image generation, remote MCP
Storage defaultstored by default for new accountsstored (30 days) unless store: false
Streamingdelta chunkstyped semantic events

Sources: migration guide, Chat Completions reference, conversation state.

Key length control is max_output_tokens, which caps reasoning plus visible text; hitting it returns status: "incomplete", possibly with no visible text, so OpenAI suggests reserving at least 25,000 tokens when starting out with reasoning models (reasoning guide). Streaming emits response.created, response.output_item.added, response.output_text.delta, response.function_call_arguments.delta, response.completed and tool-specific events (streaming guide; see streaming).

Conversation state: three options

flowchart TD
    A["New turn"] --> B{"Who holds history?"}
    B -->|"OpenAI, per response"| C["previous_response_id<br/>(stored 30 days)"]
    B -->|"OpenAI, durable object"| D["conversation: conv_...<br/>(no 30-day TTL)"]
    B -->|"You"| E["store: false<br/>replay all output items<br/>incl. encrypted reasoning"]
    C --> F["Resend instructions + tools each call<br/>Whole chain billed as input"]
    D --> F
    E --> F
  • previous_response_id chains to an earlier stored response. It does not carry instructions forward and does not save money: every earlier token in the chain is billed as input again (migration guide, conversation state). Savings come from prompt caching, not from chaining.
  • Conversations API gives a durable conversation object whose items are not subject to the 30-day response TTL (conversation state). Conversations are not ZDR-eligible (data controls).
  • Manual (stateless): store: false and replay the full output array each turn. Reasoning items then carry encrypted_content, which must be passed back in order; ZDR organizations have store: false forced (reasoning guide, data controls).

reasoning.context (auto / current_turn / all_turns; GPT-5.6 defaults to all_turns) decides whether earlier turns’ reasoning is reused; reuse works only within one model family (reasoning guide).

Compaction keeps long conversations inside the window: context_management: [{type: "compaction", compact_threshold: N}] compacts server-side when the prompt crosses N tokens, or POST /responses/compact returns a smaller window statelessly. The result is an opaque encrypted compaction item; with previous_response_id chaining you must not prune manually (compaction guide).

Background mode (background: true) runs long reasoning or agent jobs asynchronously: poll while queued/in_progress, cancel via /cancel, and resume a dropped stream with starting_after a saved sequence_number. It stores data temporarily even with store: false (about 10 minutes for ZDR projects) (background guide).

Tools

Function tools are {type: "function", name, description, parameters, strict}; calls arrive as function_call items with a call_id, and you answer with function_call_output items carrying the same call_id, passing back any reasoning items alongside (function calling; see function calling). tool_choice accepts auto, required, a named function, allowed_tools (restrict without changing the tool list, which keeps the cache warm) and none. tool_search with defer_loading loads tools on demand on gpt-5.4+, and custom tools accept free text constrained by a Lark or regex grammar (constrained decoding) (function calling).

Hosted tools run inside the request and appear as their own output items:

ToolKey controlsNotes
web_searchdomain filters (100 allow/block), user_location, search_context_size, external_web_accessurl_citation annotations; search context capped at 128K tokens; per-call fee (web search)
file_searchvector_store_ids, max_num_results, attribute filtersfile_citation annotations; results hidden unless include asks (file search)
code interpretercontainer auto or explicit, memory 1g-64gcontainer dies after 20 idle minutes; 100 RPM per org (code interpreter)
shellhosted container or local shell_callno outbound network by default; admin allowlist plus network_policy (shell)
computeraction batches in computer_callyou execute and return screenshots; safety checks need approval (computer use)
remote MCPserver_url, require_approvalsee MCP in practice

Reasoning controls

reasoning.effort ranges none, minimal, low, medium (default), high, xhigh, max depending on model; reasoning.mode: "pro" adds more work at standard token rates; reasoning.summary opts into summaries. Raw reasoning is never returned but is billed as output (output_tokens_details.reasoning_tokens). A configuration_update input item can change effort mid-conversation without breaking the cached prefix (reasoning guide; see reasoning models).

Caching, limits and pricing hooks

  • Caching is automatic by default (on GPT-5.6+ for prefixes of 1,024+ visible tokens; earlier models have model-dependent minimums). GPT-5.6+ adds prompt_cache_options: implicit (default; one automatic breakpoint at the latest message, leaving three for your own) or explicit (only your breakpoints, so no breakpoints means no caching), up to four writes per request, and ttl 30m (write 1.25x) or 12h (write 2.5x, Responses only), with reads at 0.1x input (0.05x on GPT-6.1 Sol and Terra; see also pricing). Caches live on individual machines, so above roughly 15 RPM traffic can overflow to other machines and miss; on older models prompt_cache_key steers routing, while on GPT-5.6+ routing is automatic and the key is optional (mainly for cache accounting). Changing tools, reasoning effort or output format breaks the prefix (caching guide).
  • Rate limits are RPM/TPM/RPD/TPD per tier (Free, Build $5, Launch $100, Grow $500 in cumulative purchases). TPM is charged on the larger of max_tokens and the estimated input size, so an inflated max_tokens wastes headroom (rate limits; see rate limits and retries).
  • Batch and Flex are half price, Fast mode 2x; input above 272K tokens pays a higher rate (pricing).

Side by side with Claude and Gemini

Synthesis, compiled from the three provider pages and their sources (Nov 2026):

ConcernOpenAI ResponsesAnthropic MessagesGemini Interactions
Server-side stateprevious_response_id (30 days) or conversationnone; resend historyprevious_interaction_id (55 days paid, 1 day free)
What chaining forgetsinstructionsn/atools, system_instruction, generation_config
Tool call / resultfunction_call / function_call_output items, call_idtool_use / tool_result blocks in the user turnfunction_call / function_result steps, call_id
Opaque reasoning to replayreasoning item encrypted_contentthinking block signaturethought_signature on thought steps
Effort knobreasoning.effortoutput_config.effortthinking_level
Cachingautomatic ≥1,024 tokens, optional breakpoints, 30 m / 12 h TTLopt-in: block-level cache_control or one top-level “automatic” breakpoint, ≤4 breakpoints, 5 m / 1 himplicit (≥4,096 on Gemini 3/4); explicit cachedContents on generateContent only
Server loop hand-backhosted-tool items inlinepause_turnbuilt-in tool steps inline
Files lifetimeuntil deleteduntil deletedauto-deleted after 48 h
Sampling paramstemperature 0-2, top_p documented on Chat Completions400 on version 2026-10-21silently ignored on Gemini 3.6 Flash+

Sources: conversation state, migration guide, OpenAI caching, Anthropic API reference, Anthropic caching, Anthropic thinking, Anthropic server tools, Anthropic sampling removal, Gemini Interactions, Gemini thinking, Gemini caching, Gemini Files, Gemini generateContent, Chat Completions reference.

Porting checklist (Synthesis): (1) move the system prompt to the target’s slot and re-send it every turn wherever chaining forgets it; (2) remap tool-call IDs and the role that carries results; (3) never carry reasoning artifacts across providers, and drop them when switching model family; (4) redesign cache layout, since OpenAI and Gemini cache implicitly while Claude caching is opt-in (explicit breakpoints or a top-level automatic cache_control); (5) strip sampling parameters; (6) re-upload files per provider and plan for Gemini’s 48-hour expiry; (7) re-run evals, because the same prompt behaves differently (see model routing and fallbacks).

The “OpenAI-compatible” trap

Many providers expose Chat Completions-shaped endpoints, not Responses. Anthropic’s layer ignores strict, drops prompt caching, hoists all system messages to the top, hides thinking, and is positioned for testing rather than production (Anthropic compat). Gemini’s layer is beta, covers Chat Completions, embeddings, images and batch but not Responses, and silently ignores unknown parameters (Gemini compat). Synthesis: “OpenAI-compatible” means Chat Completions syntax only; hosted tools, stored state and encrypted reasoning do not port. Use a native SDK or a gateway that translates semantics (see LLM gateways).

Failure modes and gotchas

  • Code reading choices[0].message.content breaks; not every output item is a message (migration guide).
  • Dropping reasoning or function_call items when replaying manually breaks continuity; a missing call_id match errors (migration guide).
  • Forgetting to resend instructions with previous_response_id silently loses the system prompt (migration guide).
  • A low max_output_tokens can produce an incomplete response with no visible text, all billed (reasoning guide).
  • Default storage (30 days) can conflict with data-handling commitments; set store: false deliberately (conversation state, data controls).

When to use / when not

Synthesis:

  • Use Responses for any new OpenAI integration, especially agents with tools, reasoning models, or hosted search/retrieval/code.
  • Use server-side state for prototypes and chat apps where 30-day storage is acceptable; prefer stateless replay for ZDR, multi-provider portability, and when your own store is the source of truth (see agent memory, durable agent execution).
  • Keep Chat Completions only for legacy code or third-party “OpenAI-compatible” endpoints, and accept that tool calling there is unavailable with GPT-6 Astra and GPT-6.1 Sol and limited to reasoning_effort: none on other GPT-5.4+ models.
  • Compare with the Anthropic Messages API (stateless, opt-in cache breakpoints) and the Gemini API (stateful Interactions API); the OpenAI Agents SDK builds on Responses.

Sources

Suggest changes →