Source: LoRA: Low-Rank Adaptation of Large Language Models

The 2021 Microsoft paper introducing LoRA: frozen weights plus trainable low-rank update matrices.

Contents
  • What: research paper (arXiv:2106.09685, ICLR 2022)
  • Author: Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (Microsoft)
  • Date: 2021-06-17 (v2 2021-10-16)
  • Original: arXiv:2106.09685

Summary

Hypothesises that the weight change during adaptation has low intrinsic rank, so it can be represented as a product of two thin matrices. Pre-trained weights stay frozen; only the low-rank factors are trained. The update can be merged into the weights after training, so inference costs nothing extra.

Key claims

  • Forward pass becomes h = W0·x + B·A·x, with B (d×r) and A (r×k), r ≪ min(d,k).
  • A is initialised Gaussian and B to zero, so the update starts at zero; ΔW·x is scaled by α/r, and α is set once and not tuned.
  • On GPT-3 175B: ~10,000× fewer trainable parameters, training VRAM 1.2 TB → 350 GB, checkpoint 350 GB → 35 MB (r=4, Wq and Wv only), ~25% faster training.
  • Matches or beats full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 in their experiments; very low ranks (even r=1) sufficed for some GPT-3 tasks.
  • Merging B·A into W removes inference latency, but then batching requests for different adapters in one forward pass is not straightforward.
  • Experiments adapted only attention weights; MLP adaptation was left to future work.

Feeds

PEFT and LoRA · Fine-tuning

Suggest changes →