Source: Fast and Expressive LLM Inference with RadixAttention and SGLang
LMSYS blog post explaining RadixAttention's radix-tree KV cache and LRU eviction.
Contents
- What: project blog
- Author: Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng (LMSYS)
- Date: 2024-01-17
- Original: Fast and Expressive LLM Inference with RadixAttention and SGLang
Summary
Describes how SGLang keeps KV cache after requests finish, indexed in a radix tree keyed by token sequences, so any later request with a matching prefix (system prompt, few-shot examples, chat history) skips that prefill.
Key claims
- The radix tree lives on the CPU and maps token sequences to KV tensors stored on the GPU in a paged layout with page size one token.
- Supports prefix match, insert and LRU eviction of leaf nodes; nodes split when requests diverge.
- A cache-aware scheduling policy complements LRU to raise hit rate.
- No noticeable overhead without cache hits, so it is always on; up to 5x throughput vs vLLM 0.2.5, Guidance and TGI 1.3 in their benchmarks.