Source: vLLM docs: Disaggregated Prefilling

Official, experimental vLLM feature that runs prefill and decode in separate instances linked by a KV-transfer connector.

Contents

Summary

Describes running prefill and decode as separate vLLM instances, with a connector shipping the KV cache from one to the other, and is explicit that the benefit is latency control, not throughput.

Key claims

  • Motivation: tune TTFT and ITL independently (different parallelism per phase) and control tail ITL, since prefills no longer interrupt decodes; chunked prefill helps too but chunk size is hard to tune.
  • Disaggregated prefill does not improve throughput.
  • Connectors include NixlConnector, LMCacheConnectorV1, MooncakeConnector, OffloadingConnector (KV to CPU), MultiConnector and others, chosen with --kv-transfer-config JSON (e.g. {"kv_connector":"NixlConnector","kv_role":"kv_both"}).
  • Marked experimental and subject to change.

Feeds

vLLM

Suggest changes →