Batch Processing
Asynchronous vendor batch APIs that run large request sets at about 50% off within a 24-hour window: limits, lifecycle, result handling and caching interplay across Anthropic, OpenAI and Gemini.
Contents
Batch processing means submitting many independent model requests as one asynchronous job and collecting the results later, in exchange for a large discount and separate capacity. Anthropic’s Message Batches API, OpenAI’s Batch API and Gemini’s Batch API all charge 50% of standard price and promise completion within 24 hours, usually much sooner. Each request carries a client-chosen id because results come back unordered (Anthropic, OpenAI, Gemini). Synthesis: providers can fill idle GPU capacity with work that has no latency SLO, and the discount is the price of giving up immediacy.
Lifecycle
flowchart LR
A["Build requests with unique custom_id"] --> B["Submit: inline list or uploaded JSONL"]
B --> C["Async validation + queued processing"]
C --> D{"Done or 24h window hit"}
D --> E["Results file: succeeded / errored / expired / canceled per request"]
E --> F["Join on custom_id, retry failures"]Vendor comparison (as of 2026-11)
| Anthropic Message Batches | OpenAI Batch API | Gemini Batch API | |
|---|---|---|---|
| Discount | 50% | 50% | 50% |
| Window | Most finish < 1 h; expire at 24 h | 24 h completion window (only value) | 24 h target; job expires after 48 h pending/running |
| Input | JSON list of {custom_id, params} in the create call | Uploaded JSONL: custom_id, method, url, body; one model per file | Inline list (< 20 MB) or JSONL file of GenerateContentRequest |
| Size limits | 100,000 requests or 256 MB | 50,000 requests, 200 MB file; 2,000 batches/hour | < 20 MB inline; input file up to 2 GB |
| Endpoints | Messages | Responses, Chat Completions, Embeddings, Completions, Moderations, image generations and edits | generateContent (plus embeddings) |
| Statuses | in_progress (→ canceling) → ended | validating, in_progress, finalizing, completed, failed, expired, cancelling, cancelled | JOB_STATE_PENDING … SUCCEEDED, FAILED, CANCELLED, EXPIRED |
| Per-request outcome | succeeded, errored, canceled, expired (only succeeded billed) | Output file + separate error file; expired requests get batch_expired | JSONL of responses or status objects |
| Result retention | 29 days | Files expire after 30 days by default (1 h–30 d configurable) | not specified in the docs read |
| Rate limits | Separate batch limits (HTTP + queued requests) | Separate pool; per-model queued-token limits | Higher rate limits in exchange for slower turnaround |
Sources: Anthropic, OpenAI, Gemini.
Behaviour to design around
- Unordered results. All three return results in arbitrary order, so always join on
custom_id. Anthropic restricts it to^[a-zA-Z0-9_-]{1,64}$(Anthropic, OpenAI). - Late validation. Anthropic validates each request’s params asynchronously, so a malformed request surfaces only when the whole batch ends. Dry-run one request on the synchronous API first (Anthropic).
- Partial completion. At expiry, unfinished requests fail (
expired/batch_expired). Completed results stay available and are billed (OpenAI, Anthropic). Gemini differs: a job pending or running for more than 48 hours moves toJOB_STATE_EXPIREDand produces no results (Gemini). Under heavy demand, more requests can expire (Anthropic). - Unsupported in batch (Claude).
stream: true, fast-modespeed, server-sidefallbacks, andmax_tokens: 0cache pre-warming (Anthropic). - Spend control. Anthropic batches can slightly exceed a workspace spend limit because of concurrent processing (Anthropic).
- Data retention. Anthropic batches are not Zero-Data-Retention eligible (29-day storage) (Anthropic). Check this for regulated customers. See AI governance and compliance.
Stacking discounts with caching
Batch and cache discounts multiply. Anthropic’s cache multipliers stack with the batch discount (Anthropic caching), and Gemini lists separate batch-tier cached-token rates (e.g. Gemini 3.8 Flash cached input $0.0375/MTok in batch vs $0.075 standard, through 2026; both double on 1 Jan 2027) (Gemini pricing). Gemini batch requests use explicit caches by setting cached_content per request, though the batch guide describes hits as billed “at standard context caching rates”, so confirm which rate applies on your invoice (Gemini batch). The catch: requests run concurrently and in any order, so hits are best-effort. Anthropic reports 30–98% hit rates and suggests the 1-hour TTL for shared context (Anthropic). Synthesis: to raise the hit rate, put the shared prefix in a single cache breakpoint, use the long TTL, and consider warming the cache with one synchronous request before submitting the batch.
Structured outputs work inside batches, which is the common pattern for bulk extraction (Anthropic structured outputs).
When to use / when not
- Use for offline evals and LLM-as-judge runs, dataset labelling, synthetic data for fine-tuning or distillation, bulk document processing, backfills, embedding a corpus, and nightly summarisation (Gemini, Anthropic).
- Don’t use for anything user-facing, for multi-turn agent loops where each step depends on the last (each step would be a separate batch round-trip), or when you need streaming.
- Alternatives. Synthesis: for “cheap but within minutes” work, a rate-limited worker pool on the synchronous API with prompt caching may beat batch on latency. Some vendors also sell intermediate tiers, such as Gemini’s Flex inference (Gemini pricing). See cost and latency optimization.