Rate Limits and Retries

How LLM APIs meter traffic (RPM, TPM, ITPM/OTPM, daily and spend limits, ramp limits), how to classify 429/503/529 failures, and how to build clients that survive them: backoff with jitter, retry budgets, timeouts for long generations, concurrency caps, queueing and service tiers.

Contents

Rate limits are the per-organisation (or per-project, per-subscription) ceilings an LLM provider puts on requests and tokens per unit time; retries are the client-side policy for recovering when a call is throttled, overloaded or times out. Operating LLMs at scale is mostly about these two halves fitting together: knowing exactly which meter you are about to trip, telling a transient 429 from a permanent one, and retrying in a way that does not amplify the overload. Together with provisioned throughput, this is the capacity side of enterprise LLM deployment.

What gets metered

Every major provider meters several dimensions at once, and whichever runs out first throttles you (OpenAI, Gemini).

Provider (late 2026)DimensionsScopeNotable mechanics
Anthropic Claude APIRPM, input TPM (ITPM), output TPM (OTPM), monthly spend cap per tierOrganisation, per model classToken bucket refilling continuously; cache reads don’t count toward ITPM; ramp limits on sharp increases (source)
OpenAI APIRPM, RPD, TPM, TPD, IPM; Batch queue tokensOrganisation and project, per model; some models share a poolCounted usage = max(max_tokens, estimated tokens); tiers Free/Build/Launch/Grow (source)
Gemini Developer APIRPM, input TPM, RPD (+IPM, TPD for some models), rolling 10-minute spendProject, not API keyRPD resets midnight Pacific; tiers by cumulative spend and account age (source)
Vertex AI Standard PayGoOrg-level prioritized TPM threshold per model; no RPMOrganisationShared pool: traffic above threshold may be served but is throttled first; 429 = contention (source)
Amazon Bedrock on-demandTPM, TPD, RPM quotasAccount, Region, modelReserves input + max_tokens up front; output burndown multipliers up to 15x (source)
Azure OpenAI (Foundry) StandardTPM with RPM derived from itSubscription/region pools, per deployment typeSeven quota tiers assigned automatically (source)

Four mechanics matter more than the headline numbers:

  • Bucket refill, not a minute boundary. Anthropic uses a continuously refilling token bucket, so a burst within a second can 429 even when the per-minute average is fine (source). Vertex likewise asks you to smooth spikes inside each minute (source).
  • max_tokens is reserved up front (on some providers). OpenAI counts the larger of max_tokens and the estimated prompt size (source). Bedrock deducts input + max_tokens at request start, settles at the end with output multiplied by a model-specific burndown rate (2.5x–15x for Claude models depending on version, e.g. 5x for Claude 4.7 and older, 15x for Claude 4.8), and refunds the difference. A 32K max_tokens for a 1K reply therefore cuts concurrency sharply (source). Set max_tokens close to the real output length. Anthropic’s own API is the exception: OTPM counts only tokens actually generated, so max_tokens has no rate-limit cost there (source).
  • Caching raises effective throughput. On Anthropic, cache reads don’t count toward ITPM, so 2M ITPM at an 80% hit rate means about 10M total input tokens per minute (source). Bedrock also excludes cache reads (source). See prompt caching.
  • Ramp limits. Even under your limit, a sudden jump can be throttled. Anthropic returns 429 ramp_limit_exceeded and asks for at most one step up per minute (source). OpenAI’s rule of thumb: past about 1M input TPM, grow no more than 50% per 15 minutes (source). Its Fast mode downgrades over-ramp requests to Standard speed and price, reporting service_tier: "default" in the response (source). Vertex Priority PayGo also has a ramp limit (source). Launches and model migrations should shift traffic gradually behind feature flags.

Classifying failures

The most common production bug is treating every 429 the same way. The status code alone is not enough; read the error type, error_code and headers.

SignalMeaningRetry?
429 with retry-after (Anthropic rate_limit_error, OpenAI temporary limit)Your own rate limitYes, after at least retry-after
Anthropic 429 ramp_limit_exceeded / OpenAI 429 slow_downRamped too fastYes, slower; reduce ramp rate
Anthropic 429 enforced_spend_limit_reached, no retry-afterTier monthly spend capNo, keeps failing until access resumes (source)
Anthropic 400 for a self-set spend limit, Anthropic 402 billing_error; OpenAI hard spend limit 429; quota/billing errorsBudget exhaustedNo (Anthropic, OpenAI)
Anthropic 529 overloaded_errorProvider-wide overload, not your quotaYes with backoff, or fail over
OpenAI 503 server_is_overloadedProvider overloadYes with backoff
Gemini/Vertex 429 RESOURCE_EXHAUSTEDQuota, spend window, or shared-pool contentionYes with backoff; consider the global endpoint (source)
OpenAI flex 429 Resource UnavailableNo flex capacity right now (not billed)Back off, or retry on the default tier (source)
500 api_error, 504 timeout_errorServer fault or processing timeoutYes with backoff; switch long calls to streaming
400/401/403/404/413Client errorNo (409 conflict_error: resolve the conflict first)

Headers give you pre-emptive control: anthropic-ratelimit-{requests,input-tokens,output-tokens}-{limit,remaining,reset} (source) and x-ratelimit-{limit,remaining,reset}-{requests,tokens} (source). Log the request-id on every failure for support tickets (source).

Backoff with jitter

Capped exponential backoff without randomness performs worst, because clients that failed together retry together. Adding jitter cut total calls by more than half with 100 contending clients in Brooker’s simulation (source). The variants:

  • Full jitter: sleep a uniform random time in [0, min(cap, base × 2^attempt)]. This is the variant in the owner’s tutorial code (source), and it does the least total work (source).
  • Equal jitter: keep half the window fixed and randomize the other half. Similar work, slower to finish; the weakest jittered option.
  • Decorrelated jitter: grow the random window from the previous sleep. Finishes slightly faster, at somewhat more work.

LLM-specific rules on top:

  1. Honour retry-after first, and add a small random delay to it (source).
  2. Failed requests still count toward OpenAI per-minute limits, so tight retry loops make throttling worse (source).
  3. Don’t stack retry layers. The Anthropic SDK already retries connection errors, 429s and 5xx twice by default (source); OpenAI SDKs also retry 429/503 automatically, and retry 408s twice (rate limits, flex). OpenAI’s guide says to disable SDK retries if you retry yourself. A gateway, a framework and your own loop each retrying multiplies load. Five layers × three retries can mean 243x load on the bottom dependency (source). Retry at one layer: set max_retries=0 everywhere else.
  4. Budget retries. Cap attempts and total elapsed time against the caller’s deadline (OpenAI, notebook). A client-side token bucket for retries (retry freely while tokens last, then at a fixed rate) is AWS’s alternative to circuit breakers (source).
  5. Escalate instead of retrying forever. After a bounded number of 529/503/overload errors, fail over to another region, provider or model. AWS recommends bounded retries plus multi-Region or multi-model fallback (source). See model routing and fallbacks and LLM gateways.
flowchart TD
  A["Call fails"] --> B{"Client error 4xx (not 408/429)?"}
  B -- "yes" --> X["Don't retry: fix request"]
  B -- "no" --> C{"Spend cap / quota / billing?"}
  C -- "yes" --> Y["Don't retry: alert, degrade"]
  C -- "no" --> D{"retry-after header?"}
  D -- "yes" --> E["Sleep retry-after + small jitter"]
  D -- "no" --> F["Sleep random(0, min(cap, base*2^n))"]
  E --> G{"Attempts or deadline exhausted?"}
  F --> G
  G -- "no" --> H["Retry same target"]
  G -- "yes" --> I{"Overload (529/503) or regional 429?"}
  I -- "yes" --> J["Fail over: other region / model / tier"]
  I -- "no" --> Z["Surface error to caller"]

Synthesis: the diagram combines the classification table above with the retry rules from the AWS, OpenAI and Anthropic sources.

Idempotency

Retries are only safe when repeating the call has no extra side effects. Reads usually qualify; creates often don’t (source). The standard fix is a client-generated idempotency key that the server deduplicates on, returning the saved first response for repeats (Stripe). The owner’s tutorial applies the same idea to agents: give each run a key so a duplicate trigger does not start two expensive runs (source).

Synthesis: an LLM completion call usually has no side effects beyond billing, so retrying it is safe but costs money. The danger is in the agent loop around it. A retried step that already executed a tool (sent an email, wrote a row) must not run that tool again. Put idempotency keys on side-effecting tools, and checkpoint completed steps as in durable agent execution.

Timeouts for long generations

  • Long outputs and reasoning runs can take minutes. Anthropic recommends streaming or the Message Batches API, especially for requests over 10 minutes. Networks may drop idle connections, so TCP keep-alive helps. The SDK refuses non-streaming requests expected to exceed 10 minutes (source). Errors can arrive mid-stream after a 200, so handle error events in the stream as well as status codes. See streaming.
  • OpenAI SDKs default to a 10-minute timeout and retry 408s twice; flex requests warrant longer timeouts (the docs’ examples use 15 minutes) (source).
  • Set timeouts from measured latency percentiles: pick an acceptable false-timeout rate and use the matching downstream percentile (source). Synthesis: LLM latency scales with output tokens and thinking effort, so time to first token and total time need separate budgets per route (extended thinking routes need much longer totals).

Concurrency control and queueing

Retries handle failures; concurrency limits keep you from causing them. Little’s Law sizes the steady state: in-flight requests = throughput × latency. The scarcest downstream limit, often the LLM rate limit, is the real ceiling (source). Synthesis example: 20 requests/s at 6 s average latency means about 120 concurrent calls. If each uses 3K input tokens, that is 3.6M ITPM. Check that number against the tier before launch, not after.

Practical controls, from the owner’s tutorial (source):

  • A semaphore caps in-flight calls per model at or below what the limit supports; excess callers queue, which is backpressure.
  • Bulkheads: separate pools per dependency and per tenant, so one noisy tenant or slow model can’t starve the rest.
  • A durable queue plus workers for long or bursty agent work, with per-tenant concurrency and token/spend budgets.

Synthesis: a production client typically adds a client-side token-bucket limiter keyed on (model, region). It is fed by the provider’s remaining/reset headers and charges each request the way the provider meters it: estimated input plus max_tokens on OpenAI and Bedrock, uncached input plus actual output on Anthropic. Priority classes (interactive ahead of background) decide who waits when the bucket is empty.

Service tiers as a reliability lever

Tiers trade price for queue position on the same pay-per-token API:

TierExamplesBehaviour
Priority / FastOpenAI Fast mode (service_tier="fast"/"priority"), Bedrock priority, Vertex Priority PayGo, Gemini Priority inferencePremium price, served first, ramp-limited; shares the on-demand quota on OpenAI and Bedrock; Gemini Priority has its own limits (default 0.3x standard) that also count toward interactive limits (OpenAI, Bedrock, Gemini). Anthropic’s fast mode (speed: "fast") is a speed option with dedicated rate limits and ramp limits (source)
Standarddefault everywhereBest effort, subject to 429/529
FlexOpenAI flex, Bedrock flex, Vertex Flex PayGoAbout 50% cheaper / Batch rates, slower, more throttling (OpenAI, Vertex)
BatchAnthropic Message Batches, OpenAI Batch, Gemini BatchAsynchronous, about 50% off, separate queue limits; Anthropic batches are exempt from ramp limits (source)
Reserved / provisionedsee provisioned throughputSeparate committed capacity, overflow to Standard

Anthropic’s own Priority Tier (service_tier: auto | standard_only) is marked legacy, and commitments are no longer sold as of November 2026 (existing ones run to contract end) (source). Always read the tier actually applied from the response (usage.service_tier on Anthropic, service_tier on OpenAI, ResolvedServiceTier in Bedrock CloudWatch), because requests can be downgraded silently. Move offline work to batch processing so it never competes with interactive quota.

When to use what

  • Low volume, prototype: SDK defaults (2 retries on the Anthropic SDK) are enough. Set max_tokens sensibly and log request IDs.
  • Production interactive: one retry layer with jittered backoff and a deadline, a client-side limiter, streaming for long outputs, and fallback after bounded retries. Watch the remaining-capacity headers in LLM observability.
  • Bursty or launch traffic: ramp gradually, pre-request tier or quota increases, and consider Priority tiers or provisioned capacity.
  • Offline jobs: Batch or Flex. Don’t put them on Priority.
  • Don’t: retry spend-cap errors, retry at several layers, or set max_tokens to the model maximum “just in case” on providers that reserve it (OpenAI, Bedrock).

Sources

Suggest changes →