Source: NVIDIA Dynamo (GitHub README)
NVIDIA's open-source datacenter-scale orchestration layer for disaggregated LLM serving over vLLM, SGLang and TensorRT-LLM.
Contents
- What: open-source project README (v1.6, read 2026-11-14)
- Author: NVIDIA
- Date: 2026-11-14 (accessed)
- Original: NVIDIA Dynamo (GitHub README)
Summary
Dynamo sits above inference engines and coordinates them across GPUs and nodes: separate prefill and decode pools, a router that sends requests to workers already holding matching KV cache, and a KV block manager that offloads cache to cheaper tiers.
Key claims
- Disaggregated prefill/decode pools that scale independently.
- KV-aware routing by load and KV cache overlap (README cites 2x TTFT in one benchmark).
- KV Block Manager offloads KV cache from GPU to CPU, SSD and remote/object storage.
- Backends: vLLM, SGLang, TensorRT-LLM; Apache-2.0.