Source: vLLM docs: Speculative Decoding

Official vLLM guide to its speculative decoding methods, configuration and when they help.

Contents

Summary

Lists vLLM’s speculators (EAGLE, MTP, draft model, n-gram, suffix decoding and others), the speculative_config interface, the lossless guarantee of its rejection sampler, and guidance on which methods pay off at which load.

Key claims

  • Methods include EAGLE, multi-token prediction (MTP), draft model, n-gram (prompt lookup), suffix decoding, custom speculators, dynamic speculation, adaptive verification, and a lossy FLy verification mode (rejection_sample_method=“fly”, explicitly not distribution preserving).
  • Configured via speculative_config / –speculative-config JSON with method, model and num_speculative_tokens; draft_tensor_parallel_size sets draft TP.
  • The standard rejection sampler is lossless up to hardware numerics, validated by convergence and greedy-equality tests; logprobs are not guaranteed stable.
  • Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0.
  • Targets memory-bound, low-to-medium QPS workloads; EAGLE/MTP/draft give the largest gains at low QPS, n-gram and suffix give low-to-medium gains at low QPS and medium gains at high QPS, without extra load at peak.

Feeds

Speculative decoding

Suggest changes →