Source: Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference

OSDI 2024 paper introducing chunked prefills and stall-free scheduling.

Contents

Summary

Long prefills inserted into a running batch stall every ongoing decode, causing latency spikes. Sarathi-Serve splits prefills into similar-sized chunks and fills each iteration’s token budget with decodes first and prefill chunks after, so batches stay uniform and decodes never stall.

Key claims

  • Chunked prefills split a prompt’s prefill across several iterations.
  • Stall-free scheduling admits new requests without pausing ongoing decodes.
  • Uniform batches reduce pipeline bubbles under pipeline parallelism.
  • Versus vLLM under tail-latency SLOs: 2.6x capacity (Mistral-7B, 1 A100), up to 3.7x (Yi-34B, 2 A100), up to 5.6x (Falcon-180B with PP).

Feeds

LLM inference serving

Suggest changes →