Source: LLM Inference Performance Engineering: Best Practices

Databricks/MosaicML guide to inference metrics (TTFT, TPOT, MBU) and the batch-size latency/throughput trade-off.

Contents

Summary

Defines the standard serving metrics and introduces Model Bandwidth Utilization as the right efficiency metric for memory-bound decode, then measures how batch size, tensor parallelism and quantization move latency and throughput.

Key claims

  • Latency = TTFT + TPOT x output tokens; throughput = output tokens per second across all requests.
  • MBU = achieved bandwidth / peak bandwidth, where achieved bandwidth = (weights + KV cache bytes) / TPOT. A 7B FP16 model at 14 ms TPOT moves about 1 TB/s, i.e. 50% MBU on a 2 TB/s GPU.
  • On one A100, batch 64 vs batch 1 raised latency about 4x but throughput about 14x; past some batch size decode becomes compute-bound.
  • Tensor parallelism shows diminishing latency returns (Llama2-70B: 4 to 8 GPUs gave only about 0.7x latency at small batch).
  • Output length dominates latency: 512 extra input tokens cost less than 8 extra output tokens on MPT models.
  • INT8 weights can halve the number of GPUs needed; INT8 KV cache roughly halves KV memory; quality must be checked end to end.

Feeds

LLM inference serving · KV cache · Quantization

Suggest changes →