LiteLLM
BerriAI's open-source Python SDK and self-hosted OpenAI-compatible proxy for 100+ model providers: provider-prefixed model names, a Router with fallbacks and cooldowns, Postgres-backed virtual keys and budgets, callback-based logging, and a 2026 record of supply-chain and proxy CVEs.
Contents
LiteLLM (by BerriAI) is an open-source library and server that puts 100+ model providers behind the OpenAI Chat Completions interface. It comes in two forms that share one codebase. The Python SDK (litellm.completion(), embedding(), a Router) translates OpenAI-style calls into each provider’s native API inside your process. The proxy (marketed as the “AI Gateway”) is a self-hosted HTTP server, so any OpenAI client works by changing base_url. On top of the SDK it adds virtual keys, budgets, rate limits, logging, guardrails, caching and an admin UI (docs, README). It is the most common self-hosted example of an LLM gateway, and frameworks such as Google ADK use it to reach non-native models (notebook).
SDK vs proxy
| SDK | Proxy | |
|---|---|---|
| Runs | In your Python process | As a separate service (Docker/Helm), with Postgres and Redis in production |
| Who holds provider keys | Your app (env vars) | The proxy; apps hold virtual keys |
| Routing, retries, fallbacks | Router object | Same Router, configured in config.yaml |
| Budgets, virtual keys, admin UI | No | Yes (Postgres required) |
Proxy-only hooks (async_pre_call_hook etc.) | No | Yes |
| Fits | One Python app | A platform team serving many apps or languages |
Sources: docs, proxy, logging. Enterprise features (for example per-model budgets and some logging destinations) sit under a separate commercial licence (README, budgets, logging).
Model naming
There are two layers of names, and mixing them up causes many config bugs.
- Provider strings (SDK and
litellm_params.model):<provider>/<model>, e.g.anthropic/...,bedrock/...,vertex_ai/...,ollama/...,azure/<your-deployment-name>. The prefix selects the translation code and credential env vars.openai/<anything>plusapi_basetargets any OpenAI-compatible server, such as vLLM or another local LLM server (docs). - Aliases (proxy
model_name): what clients send. Severalmodel_listentries with the samemodel_nameform a model group that the Router load-balances across.model_group_aliasmaps one alias onto another, and wildcards ("*",openai/*) pass arbitrary names through (docs).
model_list:
- model_name: chat-default # alias clients send
litellm_params:
model: azure/my-eu-deployment # provider string
api_key: os.environ/AZURE_API_KEY_EU
rpm: 600
- model_name: chat-default # same alias → same load-balancing group
litellm_params:
model: openai/my-model
api_key: os.environ/OPENAI_API_KEYSynthesis: the example above is illustrative; the model names are placeholders.
Parameter translation. OpenAI parameters are mapped per provider, and support varies by model within a provider. By default an unsupported parameter raises an exception. drop_params=True (global or per call) silently removes it instead, and get_supported_openai_params() reports what a model/provider pair accepts. Any non-OpenAI kwarg is passed straight into the provider request body. Provider errors are re-raised as OpenAI-style classes (litellm.RateLimitError, litellm.AuthenticationError, …) (docs).
Synthesis: drop_params trades a loud failure for a quiet behaviour change. A dropped tools, response_format or seed still returns a 200 with a different behaviour, which can break structured outputs or function calling on the fallback model only.
Request path through the proxy
Synthesis (assembled from the proxy, budget, router and logging docs):
flowchart TD
A["Client (OpenAI SDK, base_url = proxy)"] --> B["Auth: virtual key or master key (Postgres lookup)"]
B --> C["Budget + rpm/tpm/parallel checks (key → team → member → end user)"]
C --> D["Proxy hooks: async_pre_call_hook, guardrails"]
D --> E{"Cache hit?"}
E -- "yes" --> L["Callbacks: log + cost"]
E -- "no" --> F["Router: pick deployment in model group"]
F --> G["Translate params, inject provider key, call provider"]
G -- "error" --> H["Retry policy / cooldown deployment"]
H --> I["Fallback model group (generic, context-window, content-policy)"]
I --> F
G -- "2xx" --> J["Compute response_cost, update spend"]
J --> L
L --> K["Response + x-litellm-call-id header"]Routing, retries, fallbacks
The Router, documented in model routing and fallbacks, works the same in the SDK and the proxy (router):
- Strategies (
routing_strategy):simple-shuffle(default; weighted random byweightor rpm/tpm),latency-based-routing,usage-based-routing-v2,least-busy,cost-based-routing, or custom. The docs warn that usage-based routing adds noticeable latency from its Redis operations and recommendsimple-shufflefor production. - Retries:
num_retries, exponential backoff only on rate-limit errors, per-exception counts inretry_policy. The router sets the provider client’s ownmax_retriesto 0, so only one retry loop runs (see rate limits and retries). - Cooldowns: default
cooldown_timeis 5 s. Withoutallowed_fails, a 429 cools a deployment down at once, and so do 401/404/408 errors or a failure rate above 50% in the current minute. Settingallowed_failsswitches to failure counting. - Fallbacks: generic
fallbacks,context_window_fallbacks(useenable_pre_call_checksto skip a model whose window is too small),content_policy_fallbacks, anddefault_fallbacks.max_fallbacksdefaults to 5. - Session affinity pins a session to one deployment (idle TTL 3600 s by default), which protects prompt caching.
- Shared state: without Redis, each proxy instance keeps its own cooldowns, usage counters and affinity pins.
Keys, budgets and limits
Virtual keys (POST /key/generate) are scoped to models and carry max_budget, budget_duration, rpm_limit/tpm_limit and an owner (a user or a team). They can be rotated with a grace period (proxy). Budgets are in USD, priced per request from LiteLLM’s model cost map (proxy), and exist at these levels (budgets):
- Global:
litellm_settings.max_budgetwithbudget_duration(30s/30m/30h/30d). - Team, team member (
max_budget_in_team), internal user, virtual key (with several windows at once viabudget_limits), end user / customer (/budget/new+max_end_user_budget_id), and per model on keys or users (model_max_budget, Enterprise). - A key that belongs to a team is checked against the team and team-member budgets, not its owner’s personal budget. v1.94.0 briefly enforced the owner’s budget; v1.95.0 reverted it.
- Windows reset on calendar boundaries in UTC (
24hat midnight,7don Monday,30don the 1st). A reset job runs about every 10 minutes by default. - Rate limits:
rpm_limit,tpm_limitandmax_parallel_requestson keys, users and teams, plusmodel_rpm_limit/model_tpm_limit; key values override team values.token_rate_limit_typeselects input, output or total tokens.
Failure semantics matter more than the knobs:
- No Postgres → budgets fail open. Requests keep flowing (budgets).
fail_closed_budget_enforcementandfail_closed_rate_limit_enforcement(503 when Redis cannot confirm counters) are opt-in (budgets).max_parallel_requestsreturns an immediate 429 and does not queue, and per-deployment caps are per process (router).- Models priced at zero cost skip budget checks entirely, so a self-hosted model with no cost entry is effectively unbudgeted (budgets).
Logging callbacks
All observability goes through callbacks (logging):
- Named integrations:
success_callback/failure_callback/callbackslists such as["langfuse"],["otel"],["s3_v2"], Datadog, LangSmith, Arize, MLflow and more. In the proxy these sit underlitellm_settings. See Langfuse and OpenTelemetry GenAI. CustomLoggersubclasses registered vialitellm.callbacks(SDK) or a dotted path to an instance (proxy). Request-level hooks:log_pre_api_call,async_log_success_event,async_log_failure_event. Per-attempt hooks (async_*_deployment_hook) fire once per retry or fallback attempt. Proxy-only hooks (async_pre_call_hook,async_post_call_success_hook) can rewrite requests and responses, the gateway form of agent hooks and middleware.- Payload: callback kwargs include
response_cost,cache_hitand your metadata. Success and failure events carry astandard_logging_object(StandardLoggingPayload), which may be absent on intermediate streaming events. - Privacy:
turn_off_message_loggingkeeps spend and metadata but drops prompts and responses.redact_user_api_key_infostrips key identity for some integrations. Clients can sendno-log: trueunlessglobal_disable_no_log_paramis set. This interacts with data retention and residency and PII handling. - Loss under load: the S3 logger queues per worker and drops entries older than 3600 s or past 50,000 queued items by default.
Synthesis: treat LiteLLM logs as cost and audit telemetry, not a guaranteed record. Put tracing that must not be lost in LLM observability tooling with its own delivery guarantees, and remember that per-attempt hooks are where retries and fallbacks become visible.
Security record
A gateway concentrates every provider key in one place, so its CVEs are tier-0 incidents (CSA). The 2026 record:
| Date (2026) | Issue | Versions | Notes |
|---|---|---|---|
| Mar 24 | PyPI supply-chain compromise: 1.82.7 and 1.82.8 shipped a credential stealer | 1.82.7, 1.82.8 | Suspected entry via Trivy in CI; live about 40 min; pinned Docker image unaffected; v1.83.0 rebuilt, cosign-signed images (advisory) |
| Apr 19-20 | CVE-2026-42208: pre-auth SQL injection via the Authorization: Bearer header during key verification, CVSS 9.3 | 1.81.16-1.83.6, fixed 1.83.7 | Exploitation seen about 36 h after advisory indexing; payloads targeted key, credential and config tables (LiteLLM, CSA); CISA KEV alert dated May 8 (Security Affairs) |
| Jun 8 (KEV) | CVE-2026-42271: MCP server test endpoints ran caller-supplied commands as subprocesses with no role check, CVSS v4 8.7 | 1.74.2-1.83.6, fixed 1.83.7 | Any valid API key could run commands on the host (Security Affairs) |
| May-Nov | GitHub advisories: Host-header auth bypass, salt-key reuse giving admin escalation, unauthenticated file read leaking the Enterprise master key (all critical); SSRF and credential exfiltration via request-body routing params and user_config; local file reads via credential/OIDC params | various | First page of the advisory list only (advisories) |
Synthesis: the pattern is consistent. Request-body fields that the proxy trusts (routing parameters, user_config, credential file paths, MCP server configs) give a caller with an ordinary key server-side fetches, file reads or execution. Exposed admin and test endpoints widen that. This maps onto OWASP Top 10 for LLM supply-chain and excessive-agency risks and the broader LLM security risks. Exploitation of 42208 began within days, so patch windows measured in weeks are too slow.
Comparisons
- vs hosted aggregators (OpenRouter, Vercel AI Gateway): LiteLLM keeps the data path and keys in your VPC. You pay for that by running Postgres and Redis and owning upgrades and CVE response (LLM gateways).
- vs Kong / Envoy AI gateways: those plug into existing API-gateway and Kubernetes practice. LiteLLM is Python-native, has the broadest provider translation and the richest LLM-specific budget hierarchy, and is the easiest to embed as an SDK (LLM gateways).
- vs a framework abstraction (Vercel AI SDK, LangChain model classes): those normalise providers inside one language runtime. LiteLLM’s proxy normalises them at the network layer for every language.
Synthesis: these comparisons summarise the gateway page plus this one; they are not vendor benchmarks.
Practical guidance
Synthesis, grounded in the sources above:
- Choose the mode deliberately. Use the SDK for a single Python service. Use the proxy once more than one team or language needs shared keys, budgets or audit.
- Pin and verify. Use
-stableimages, which the README says pass 12-hour load tests, pinned by digest. Do notpip install litellmunpinned in CI, as the March 2026 incident showed (advisory, README). Stay at or above the vendor’s current recommended floor (v1.83.10-stable at the time of the 42208 post). - Never expose the proxy or admin UI to the internet without a front door. Put it behind a VPN or API gateway. Give provider keys least privilege and per-project scope, so that a database dump does not equal an org-wide compromise (CSA).
- Run Postgres and Redis in production and decide fail-open vs fail-closed explicitly for budgets and rate limits.
- Use aliases, not provider strings, in apps. Clients send
model_name, and the platform team swaps deployments underneath. - Avoid
drop_paramsfor capability-bearing params, or verify that every fallback model supportstoolsandresponse_formatbefore adding it to a chain. - One retry layer. Set client-side
max_retries=0when calling the proxy, because the proxy already retries (router). - Keep semantic caching off agentic traffic (proxy; see semantic caching).
- Turn off message logging by default and enable it per team where policy allows (logging).
Sources
- LiteLLM docs: getting started, input params, config.yaml
- LiteLLM GitHub README
- LiteLLM Router docs
- LiteLLM proxy docs
- LiteLLM budgets and rate limits docs
- LiteLLM logging and callbacks docs
- LiteLLM security update, March 2026
- LiteLLM blog: CVE-2026-42208
- CSA research note on CVE-2026-42208
- Security Affairs: KEV listing of CVE-2026-42208
- Security Affairs: KEV listing of CVE-2026-42271
- LiteLLM GitHub security advisories
- ADK notebook