Source: Portkey AI Gateway docs (fallbacks, retries, simple and semantic cache)
Portkey's gateway config reference: ordered fallback targets on status codes, retries with exponential backoff or Retry-After, and exact or semantic caching with documented limits.
Contents
- What: official documentation (“Fallbacks”, “Automatic Retries”, “Cache (Simple & Semantic)”)
- Publisher: Portkey
- Date: living docs, read 2026-11-14
- Original: Fallbacks, Retries, Cache
Summary
Portkey routes requests through a JSON “config” attached to the request. Configs declare a strategy (fallback, load balance, conditional), targets, retries and cache, and can nest.
Key claims
strategy.mode: "fallback"triestargetsin order;on_status_codes(e.g.[429, 503]) limits which failures trigger fallback, otherwise any non-2xx does. Targets can themselves be load balancers, conditional routers or fallbacks.retry.attemptsup to 5; default retried codes 429, 500, 502, 503, 504, 529; settingon_status_codesreplaces them. Backoff is exponential (1, 2, 4, 8, 16 s).use_retry_after_headershonoursretry-after-ms,x-ms-retry-after-msorretry-after; total wait is capped at 60 s, and a single Retry-After over 60 s fails immediately. Response headerx-portkey-retry-attempt-countreports the outcome.- Simple cache matches the full body exactly. Semantic cache matches only user text by cosine similarity (default threshold 0.95) while model and parameters must match exactly; the system prompt is ignored for matching.
- Semantic cache limits: under 8,191 tokens and at most 4 messages; needs a vector DB and embedding provider; Enterprise-only.
max_age60 s to 90 days (default 7 days). Namespaces partition the cache;x-portkey-cache-force-refreshbypasses it.