Source: Towards Deterministic Inference in SGLang and Reproducible RL Training (LMSYS)

SGLang team's post adding a deterministic-inference mode built on batch-invariant kernels, with seeded sampling, CUDA-graph support, overhead benchmarks and an RL reproducibility recipe.

Contents

Summary

Builds on Thinking Machines’ batch-invariant operators to give SGLang a deterministic mode, extends it to chunked prefill, CUDA graphs and seeded non-greedy sampling, measures the cost, and uses it to make RL training runs reproducible end to end.

Key claims

  • Enabled with --enable-deterministic-inference plus an attention backend of flashinfer, fa3 or triton (SGLang 0.5.3+).
  • Attention uses a fixed split-KV size; chunked-prefill boundaries are aligned to multiples of that size; FA3 fixes the number of splits to 1.
  • Per-request seeds make sampling at temperature > 0 deterministic: seeded Gumbel noise replaces torch.multinomial.
  • In 50-trial tests on Qwen3-8B, normal mode produced several distinct outputs; deterministic mode produced one in every test.
  • Overhead on Qwen3-8B (H200, 256 requests): roughly +24% to +55% latency depending on backend and length; FlashInfer and FA3 average 34.35% slowdown, versus 61.5% reported by Thinking Machines. CUDA graphs give about 2.8x speedup within deterministic mode.
  • Limits at publication: dense models only (MoE planned), deterministic only at TP1/TP2, radix cache support incomplete; recommended mainly for debugging and reproducibility.
  • Reproducible RL training also needed deterministic training settings (e.g. Megatron deterministic mode, NCCL_ALGO=Ring, torch.use_deterministic_algorithms(True)).

Feeds

LLM non-determinism

Suggest changes →