Source: DeepSeek-V3/R1 Inference System Overview

DeepSeek's production serving write-up: cross-node expert parallelism with prefill/decode disaggregation, compute-communication overlap, EP load balancing, and per-node throughput and cost figures.

Contents

Summary

Explains why DeepSeek serves V3/R1 with very wide expert parallelism and how. Because only 8 of 256 experts fire per layer, each expert needs a very large global batch to be compute-efficient; spreading experts over many GPUs both enlarges the effective batch per expert and reduces the expert memory each GPU must read per step.

Key claims

  • Prefill: routed experts EP32, attention and shared experts DP32, 4 nodes, 32 redundant experts (9 routed + 1 shared per GPU).
  • Decode: routed experts EP144, attention/shared DP144, 18 nodes, 32 redundant experts (2 routed + 1 shared per GPU).
  • Communication is hidden by splitting the batch into two micro-batches (prefill) or a 5-stage pipeline (decode).
  • Three balancers: prefill (attention compute, tokens per GPU), decode (KV cache and requests per GPU), and expert-parallel (minimise the max dispatch load on any GPU).
  • Matmuls and dispatch in FP8; core MLA and combine in BF16.
  • Per H800 node: ~73.7k input tokens/s (incl. cache hits) and ~14.8k output tokens/s; 56.3% of input tokens hit the on-disk KV cache.
  • At $2/GPU-hour the fleet cost ~$87k/day against a theoretical $562k/day if all tokens were billed at R1 prices (545% theoretical margin, lower in practice).

Feeds

Mixture of experts

Suggest changes →