Source: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving

OSDI 2024 paper arguing for running prefill and decode on separate GPUs, optimizing goodput under TTFT/TPOT SLOs.

Contents

Summary

Colocating prefill and decode on the same GPUs makes them interfere and couples their parallelism choices. DistServe puts each phase on its own GPU pool, picks parallelism per phase for the application’s TTFT and TPOT targets, and places phases with cluster bandwidth in mind to move the KV cache cheaply.

Key claims

  • TTFT measures prefill latency; TPOT measures time between output tokens during decode.
  • Colocated systems suffer strong prefill-decode interference and must overprovision or favor one metric.
  • Goodput = maximum request rate per GPU that meets both TTFT and TPOT constraints.
  • Reported 7.4x more requests served or 12.6x tighter SLO than state-of-the-art systems, with over 90% of requests meeting latency targets.

Feeds

LLM inference serving

Suggest changes →