Source: Hugging Face Hub docs: GGUF

Hub documentation for the GGUF file format and its quantization types with bits-per-weight.

Contents

Summary

GGUF is the binary single-file model format of GGML/llama.cpp, holding tensors plus standardized metadata (unlike tensor-only safetensors). The page tabulates its quantization types and their effective bits per weight.

Key claims

  • GGUF was created by Georgi Gerganov (llama.cpp) and is used by llama.cpp, Ollama, LM Studio and GPT4All.
  • K-quants use super-blocks: Q4_K is 4.5 bpw, Q5_K 5.5 bpw, Q6_K 6.5625 bpw, Q3_K 3.4375 bpw, Q2_K 2.625 bpw.
  • I-quants use an importance matrix: IQ4_XS 4.25 bpw, IQ3_XXS 3.06, IQ2_XXS 2.06, IQ1_S 1.56 bpw.
  • Legacy Q4_0/Q8_0 etc. use 32-weight blocks with round-to-nearest; MXFP4 and ternary TQ types are also defined.

Feeds

Quantization

Suggest changes →