Source: SGLang: Efficient Execution of Structured Language Model Programs

Paper introducing SGLang, RadixAttention and compressed-FSM constrained decoding.

Contents

Summary

SGLang pairs a frontend language for multi-call LLM programs with a runtime that exploits their structure: RadixAttention reuses KV cache across calls that share prefixes, and compressed finite state machines speed up structured (e.g. JSON) decoding.

Key claims

  • RadixAttention provides automatic KV cache reuse across calls.
  • Compressed finite state machines decode structured output faster.
  • Up to 6.4x throughput over state-of-the-art systems on agent control, reasoning, few-shot, JSON decoding, RAG and multi-turn chat workloads.

Feeds

LLM inference serving · KV cache

Suggest changes →