Source: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

MLSys 2024 best paper: protect salient weight channels, identified from activations, via equivalent scaling.

Contents

Summary

AWQ observes that about 1% of weights matter disproportionately and that activation magnitudes, not weight magnitudes, reveal which. It scales those channels up before quantization (with the inverse scale folded elsewhere), so no mixed precision and no backpropagation are needed.

Key claims

  • Salient weights (~1%) are identified from activation statistics.
  • Per-channel scaling is a mathematically equivalent transform, avoiding mixed-precision kernels.
  • No backpropagation or reconstruction, so it overfits the calibration set less and generalizes across domains and modalities.
  • TinyChat runtime is over 3x faster than Hugging Face FP16 and runs Llama-2 70B on mobile GPUs.

Feeds

Quantization

Suggest changes →