Quantization

Storing and computing LLM weights, activations and KV cache in fewer bits (FP8, INT8, INT4): how the methods work (GPTQ, AWQ, SmoothQuant, LLM.int8, GGUF), what they cost in accuracy, and which to use where.

Contents

Quantization maps a model’s numbers (weights, and optionally activations and the KV cache) from 16- or 32-bit floats to lower-precision formats such as FP8, INT8 or INT4. LLM decoding is memory-bandwidth bound, so fewer bytes per weight means less data moved per token. That gives more tokens per second, fewer GPUs, and more room for KV cache. The price is some approximation error, which grows unevenly as bits go down (Notebook tutorial, NVIDIA).

Mechanics

Affine mapping. A real value r is represented by an integer q using a scale S and a zero-point Z: q = round(r/S) + Z, r ≈ S(q − Z). Symmetric schemes set Z = 0. The granularity of S trades accuracy against overhead: per-tensor, per-channel (per output row), or per-group (e.g. every 32-256 weights). GGUF’s K-quants, for example, use blocks of 16-32 weights grouped into super-blocks, each with its own scale and min (Notebook, GGUF docs).

Outliers are the enemy. One large value stretches the range, coarsens S, and costs precision for every ordinary value. Weights are fairly well behaved. Activations contain systematic outlier channels, which is why activation quantization is harder (Notebook, NVIDIA, LLM.int8()).

Naming: WxAy. W4A16 means 4-bit weights with 16-bit activations (“weight-only”). The weights are dequantized on the fly inside the matmul kernel, which saves memory and bandwidth but runs the arithmetic at 16-bit. W8A8 quantizes both, so the matmul itself runs on INT8 or FP8 tensor cores and also gains compute throughput (vLLM docs, SmoothQuant). Synthesis: weight-only methods help most in memory-bound decode at small batch sizes. W8A8 helps more in compute-bound prefill and large batches.

PTQ vs QAT. Post-training quantization needs no retraining, only a small calibration set for static ranges or methods like GPTQ/AWQ. Quantization-aware training simulates rounding during training and holds up better at very low bit widths, at the cost of a training run (Notebook).

Formats and methods

MethodSchemeHow it worksNotes
FP8 (E4M3 / E5M2)W8A8 float, also KV8-bit floats: E4M3 (more precision) or E5M2 (more range). Matched 16-bit quality in training up to 175B; works for PTQ where INT8 failedNeeds Ada/Hopper+ for native FP8 math (FP8 paper, vLLM)
LLM.int8() (bitsandbytes)W8A8 with 16-bit outliersVector-wise INT8, with outlier feature dimensions (<0.1% of values) routed through an FP16 matmulHalves memory with no degradation up to 175B; easy, but not the fastest (LLM.int8())
SmoothQuantW8A8 INT8Divides activations by per-channel factors and multiplies weights by them, moving outliers into the easier-to-quantize weightsUp to 1.56x faster, 2x less memory, negligible loss (SmoothQuant)
GPTQW4A16 / W3A16Layer-by-layer rounding, using approximate second-order (Hessian) information from calibration data to compensate each rounding error in the remaining weights175B model in about 4 GPU-hours; 3.25x (A100) / 4.5x (A6000) faster than FP16 (GPTQ)
AWQW4A16Finds the ~1% salient weight channels from activation statistics and scales them up before rounding (an equivalent transform, no mixed precision, no backprop)Generalizes well across domains; MLSys 2024 best paper (AWQ)
GGUF (llama.cpp)Many: Q8_0 … Q2_K, IQ1-IQ4, MXFP4Single-file format with tensors plus metadata; block-wise K-quants (Q4_K = 4.5 bpw, Q5_K = 5.5, Q6_K ≈ 6.56) and importance-matrix I-quants (IQ4_XS = 4.25 bpw, IQ2_XXS = 2.06)The local/CPU/Apple ecosystem (llama.cpp, Ollama, LM Studio) (GGUF docs)
NF4 (QLoRA)4-bit weights for fine-tuningFrozen 4-bit base with LoRA adapters trained on topFor training on small GPUs; see PEFT and LoRA (Notebook)
FP4 (NVFP4/MXFP4)4-bit floatSupported by TensorRT-LLM on Blackwell (B200); MXFP4 is also a GGUF typeNewest tier as of 2026 (TensorRT-LLM, GGUF docs)

Note the bits-per-weight: the stored scales and mins add overhead, so “4-bit” GGUF Q4_K is really 4.5 bpw (GGUF docs).

Choosing a method

flowchart TD
  A["Where will it run?"] -->|"Laptop / CPU / Apple / edge"| G["GGUF via llama.cpp or Ollama: Q4_K / Q5_K / Q8_0"]
  A -->|"Datacenter GPU"| B{"GPU generation?"}
  B -->|"Hopper / Ada / Blackwell"| F["FP8 W8A8 first (+ FP8 KV); FP4 on Blackwell"]
  B -->|"Ampere / Turing"| C{"Memory-bound and must fit fewer GPUs?"}
  C -->|"yes"| W["INT4 W4A16: AWQ or GPTQ (Marlin kernels)"]
  C -->|"no, need compute speed"| S["INT8 W8A8 (SmoothQuant-style)"]
  F --> E["Evaluate on your own tasks"]
  W --> E
  S --> E
  G --> E

Synthesis: this decision tree combines the hardware matrix in the vLLM docs, the FP8/FP4 support in TensorRT-LLM and the method trade-offs above. It is not a recipe taken from any single source.

Engine support (as of 2026-11): vLLM loads AWQ, GPTQ, bitsandbytes, GGUF, FP8, INT8 W8A8 and LLM Compressor / ModelOpt / Quark checkpoints. AWQ needs Turing or newer, Marlin kernels need Turing/Ampere or newer, and FP8 W8A8 needs Ada/Hopper (or AMD) (vLLM docs). TensorRT-LLM targets FP8 on Hopper and FP4 on Blackwell (TensorRT-LLM). TGI supported bitsandbytes and GPTQ but is now in maintenance mode (TGI docs).

Sizing impact

Synthesis: weight memory ≈ parameters × bits / 8. A 70B-class model is about 140 GB in BF16, about 70 GB in FP8/INT8, and about 35-40 GB at 4-bit (plus scale overhead). Databricks notes that going from 16-bit to 8-bit can halve the GPU count for Llama2-70B, and 4-bit can bring models onto consumer hardware (Databricks). On a fixed GPU count, the freed memory becomes KV cache and therefore concurrency (see the worked example in KV cache). The KV cache can itself be stored in FP8, which halves it relative to BF16. Calibrate its scales (vLLM KV docs).

Limitations and failure modes

  • Uneven degradation. 8-bit is usually near-lossless and 4-bit is often a good trade. Below that, quality drops faster, and reasoning and rare knowledge suffer more than perplexity shows (Notebook). Results vary by architecture and size, so evaluate the whole system on your own tasks (Databricks). See LLM evaluation.
  • Smaller ≠ faster unless kernels and hardware support the format. An INT4 model on unsupported hardware may be no quicker (Notebook). Weight-only INT4 still computes in 16-bit (vLLM docs).
  • Calibration overfitting. Reconstruction-based methods can overfit the calibration set. AWQ avoids backprop and reconstruction partly for this reason (AWQ). Use calibration data that looks like production traffic.
  • Uncalibrated FP8 KV (scales of 1.0) can cost accuracy (vLLM KV docs).
  • Order matters in a compression pipeline: distill, then prune and fine-tune, then quantize last (Notebook). See knowledge distillation.

When to use / when not

  • Use FP8 by default on Hopper-class GPUs for serving. Use INT4 AWQ/GPTQ to fit a model on fewer or older GPUs, and GGUF for local LLMs. Use FP8 KV when KV-bound.
  • Avoid or test hard when a task is sensitive (math, code, long-tail facts, multilingual) or when going below about 4 bits. Also test hard when the provider’s published quantized checkpoint has not been evaluated on your domain.
  • Synthesis: if a model fits comfortably at BF16 and you are compute-bound at large batch sizes, weight-only INT4 may not help. Measure.

Sources

Suggest changes →