Batch Processing

Asynchronous vendor batch APIs that run large request sets at about 50% off within a 24-hour window: limits, lifecycle, result handling and caching interplay across Anthropic, OpenAI and Gemini.

Contents

Batch processing means submitting many independent model requests as one asynchronous job and collecting the results later, in exchange for a large discount and separate capacity. Anthropic’s Message Batches API, OpenAI’s Batch API and Gemini’s Batch API all charge 50% of standard price and promise completion within 24 hours, usually much sooner. Each request carries a client-chosen id because results come back unordered (Anthropic, OpenAI, Gemini). Synthesis: providers can fill idle GPU capacity with work that has no latency SLO, and the discount is the price of giving up immediacy.

Lifecycle

flowchart LR
    A["Build requests with unique custom_id"] --> B["Submit: inline list or uploaded JSONL"]
    B --> C["Async validation + queued processing"]
    C --> D{"Done or 24h window hit"}
    D --> E["Results file: succeeded / errored / expired / canceled per request"]
    E --> F["Join on custom_id, retry failures"]

Vendor comparison (as of 2026-11)

Anthropic Message BatchesOpenAI Batch APIGemini Batch API
Discount50%50%50%
WindowMost finish < 1 h; expire at 24 h24 h completion window (only value)24 h target; job expires after 48 h pending/running
InputJSON list of {custom_id, params} in the create callUploaded JSONL: custom_id, method, url, body; one model per fileInline list (< 20 MB) or JSONL file of GenerateContentRequest
Size limits100,000 requests or 256 MB50,000 requests, 200 MB file; 2,000 batches/hour< 20 MB inline; input file up to 2 GB
EndpointsMessagesResponses, Chat Completions, Embeddings, Completions, Moderations, image generations and editsgenerateContent (plus embeddings)
Statusesin_progress (→ canceling) → endedvalidating, in_progress, finalizing, completed, failed, expired, cancelling, cancelledJOB_STATE_PENDING … SUCCEEDED, FAILED, CANCELLED, EXPIRED
Per-request outcomesucceeded, errored, canceled, expired (only succeeded billed)Output file + separate error file; expired requests get batch_expiredJSONL of responses or status objects
Result retention29 daysFiles expire after 30 days by default (1 h–30 d configurable)not specified in the docs read
Rate limitsSeparate batch limits (HTTP + queued requests)Separate pool; per-model queued-token limitsHigher rate limits in exchange for slower turnaround

Sources: Anthropic, OpenAI, Gemini.

Behaviour to design around

  • Unordered results. All three return results in arbitrary order, so always join on custom_id. Anthropic restricts it to ^[a-zA-Z0-9_-]{1,64}$ (Anthropic, OpenAI).
  • Late validation. Anthropic validates each request’s params asynchronously, so a malformed request surfaces only when the whole batch ends. Dry-run one request on the synchronous API first (Anthropic).
  • Partial completion. At expiry, unfinished requests fail (expired / batch_expired). Completed results stay available and are billed (OpenAI, Anthropic). Gemini differs: a job pending or running for more than 48 hours moves to JOB_STATE_EXPIRED and produces no results (Gemini). Under heavy demand, more requests can expire (Anthropic).
  • Unsupported in batch (Claude). stream: true, fast-mode speed, server-side fallbacks, and max_tokens: 0 cache pre-warming (Anthropic).
  • Spend control. Anthropic batches can slightly exceed a workspace spend limit because of concurrent processing (Anthropic).
  • Data retention. Anthropic batches are not Zero-Data-Retention eligible (29-day storage) (Anthropic). Check this for regulated customers. See AI governance and compliance.

Stacking discounts with caching

Batch and cache discounts multiply. Anthropic’s cache multipliers stack with the batch discount (Anthropic caching), and Gemini lists separate batch-tier cached-token rates (e.g. Gemini 3.8 Flash cached input $0.0375/MTok in batch vs $0.075 standard, through 2026; both double on 1 Jan 2027) (Gemini pricing). Gemini batch requests use explicit caches by setting cached_content per request, though the batch guide describes hits as billed “at standard context caching rates”, so confirm which rate applies on your invoice (Gemini batch). The catch: requests run concurrently and in any order, so hits are best-effort. Anthropic reports 30–98% hit rates and suggests the 1-hour TTL for shared context (Anthropic). Synthesis: to raise the hit rate, put the shared prefix in a single cache breakpoint, use the long TTL, and consider warming the cache with one synchronous request before submitting the batch.

Structured outputs work inside batches, which is the common pattern for bulk extraction (Anthropic structured outputs).

When to use / when not

Sources

Suggest changes →