Source: vLLM docs: Batch Invariance
vLLM's documentation for its beta batch-invariant mode (VLLM_BATCH_INVARIANT=1): supported hardware, tested models, tensor-parallel caveats and cost.
Contents
- What: official documentation (latest/developer-preview docs, page footer dated 2026-11-04)
- Author: vLLM project
- Date: accessed 2026-11-14
- Original: vLLM docs: Batch Invariance
Summary
Describes vLLM’s opt-in mode that makes a request’s output independent of batch size and of the order in which requests are batched. Intended for debugging, RL rollouts and systems that need reproducible results. The feature is labelled beta, with open work tracked in a GitHub issue.
Key claims
- Enabled with the environment variable
VLLM_BATCH_INVARIANT=1, for bothvllm serveand the offlineLLMclass. - Hardware: NVIDIA GPUs with compute capability 8.0+, AMD Instinct (gfx90a, gfx942, gfx950), and Intel XPUs via Triton (which also need the
TRITON_ATTNattention backend). - Validated models include DeepSeek V3/R1, Qwen3 (dense, MoE, VL), Qwen2.5, Llama 3.x, GPT-OSS, Mistral-7B-v0.3, GLM-4, Gemma 2 and others; other models may work but are unvalidated.
- Under tensor parallelism, vLLM keeps custom all-reduce with a fixed reduction order and disables FlashInfer, AITER and QuickReduce all-reduce.
- Known gap: all-reduces whose size is not a multiple of 16 bytes can switch between NCCL and custom all-reduce depending on batch size, which breaks invariance.
- On Ada, Hopper and Blackwell, the mode runs without
torch.compile, using breakable CUDA graphs. - The mode can be slower than the default; the docs frame this as a deliberate reproducibility-for-speed trade.