Source: Latency optimization (OpenAI guide)
OpenAI's seven principles for cutting LLM application latency, from smaller models and fewer output tokens to parallelism, streaming and not calling an LLM at all.
Contents
- What: official API documentation guide
- Publisher: OpenAI
- Date: accessed 2026-11-14
- Original: Latency optimization
Summary
Organises latency work into seven principles and illustrates them with a customer-service example. The central point is that output tokens dominate latency, so generating fewer of them matters far more than trimming the prompt.
Key claims
- Process tokens faster: model size is the main factor; a smaller model can hold quality with a longer prompt, more examples, or fine-tuning/distillation. Predicted outputs speed up generation when most of the output is known beforehand (e.g. code edits).
- Generate fewer tokens: generation is usually the slowest step; cutting output by 50% can cut latency by roughly 50%. Shorten JSON field and function names; use
max_tokensor stop sequences. - Use fewer input tokens: halving the prompt may improve latency by only 1-5%, so it matters mainly for very large contexts; put dynamic content late to keep the shared prefix cacheable.
- Make fewer requests: each round trip adds latency; combine sequential steps into one call returning a structured object.
- Parallelise independent steps; for dependent ones, speculatively start the likely next step and cancel on a mismatch (works well for classification such as moderation).
- Make users wait less: streaming is the most effective perceived-latency technique; also process output in chunks and show steps or progress.
- Don’t default to an LLM: hard-code fixed responses, pre-compute when inputs are constrained, use UI components, or classic caching and lookups.