PEFT and LoRA
Parameter-efficient fine-tuning: freeze the base model and train a small adapter. How LoRA, QLoRA, DoRA and rsLoRA work, their key hyperparameters, and when they match full fine-tuning.
Contents
Parameter-efficient fine-tuning (PEFT) adapts a pre-trained model by training a small number of new or selected parameters while the base weights stay frozen. LoRA (Low-Rank Adaptation) is the dominant PEFT method. It represents the weight update of each targeted matrix as the product of two thin, trainable matrices, which you can keep as a swappable adapter or merge into the base weights after training (LoRA). PEFT is what makes fine-tuning cheap enough for most teams: a small adapter instead of the whole model (notebook).
How LoRA works
LoRA rests on the hypothesis that the weight change needed for adaptation has a low “intrinsic rank”. For a frozen weight W0 (d×k):
h = W0·x + (α/r) · B·A·x B: d×r, A: r×k, r ≪ min(d, k)- Initialisation. A is random (Gaussian in the paper, Kaiming-uniform in PEFT) and B is zero, so the adapter starts as a no-op and training begins exactly at the base model (LoRA, PEFT).
- Scaling α/r. This keeps the update size roughly stable as r changes. The authors set α once and didn’t tune it, since with Adam tuning α is roughly the same as tuning the learning rate (LoRA). rsLoRA scales by α/√r instead, which keeps higher ranks stable (PEFT).
- No inference latency once merged.
W = W0 + (α/r)BAfolds back into the base weights (merge_and_unload()in PEFT). The catch: once merged, you can’t easily batch requests for different adapters in one forward pass. Keeping adapters unmerged lets you choose one per request, at some cost (LoRA, PEFT). - Savings on GPT-3 175B. About 10,000× fewer trainable parameters. Training VRAM fell from 1.2 TB to 350 GB, because frozen weights need no optimizer state. Checkpoints shrank from 350 GB to 35 MB. Training ran about 25% faster. A hundred task adapters take about 354 GB instead of 35 TB (LoRA).
flowchart LR X["input x"] --> W["Frozen W0 (d×k)"] X --> A["A (r×k), trainable"] A --> B["B (d×r), trainable, init 0"] B --> S["scale α/r"] W --> P["+"] S --> P P --> H["output h"] P -.->|"after training"| M["Merge: W = W0 + (α/r)·BA"]
QLoRA: LoRA on a 4-bit base
QLoRA backpropagates through a frozen 4-bit base model into LoRA adapters that train in higher precision. This let a 65B model fine-tune on one 48 GB GPU, and the authors claim parity with 16-bit fine-tuning (QLoRA). It adds three tricks:
- NF4 (4-bit NormalFloat): a 4-bit data type designed for normally distributed weights.
- Double quantization: the quantization constants are quantized too.
- Paged optimizers: absorb memory spikes.
In practice, QLoRA uses about 4× less VRAM than 16-bit LoRA and is slightly slower and slightly less accurate (Unsloth). In TRL you pass a BitsAndBytesConfig together with a peft_config (TRL). PEFT’s loftq init reduces quantization error for quantized bases (PEFT). See quantization.
Variants and options (PEFT library)
| Option | What it does | Use when |
|---|---|---|
target_modules="all-linear" | Adapts every linear layer, not just attention q/v (the PEFT default) | Almost always (see below) |
use_rslora=True | Scales by α/√r | Higher ranks (64+) |
use_dora=True | Splits the update into magnitude and direction (LoRA) | Low ranks; adds overhead unless merged |
init_lora_weights="pissa" / "olora" / "eva" | SVD-, QR- or data-driven initialisation | Faster convergence experiments |
init_lora_weights="loftq" | Initialises to offset quantization error | Quantized bases |
modules_to_save=[...] | Fully trains extra modules, e.g. a classifier head or lm_head | New heads or vocabularies |
add_weighted_adapter(...) | Combines adapters (linear, svd, cat, TIES, DARE) | Merging task adapters |
Source: PEFT.
Hyperparameters that matter
| Knob | Starting value | Evidence |
|---|---|---|
| Target modules | All linear layers: q, k, v, o, gate, up, down | Attention-only LoRA underperforms even at matched parameter count (LoRA Without Regret, Unsloth) |
| Rank r | 16–32 (8–128 range) | Higher ranks cost memory and can overfit (Unsloth); in RL even r=1 matches full fine-tuning (LoRA Without Regret) |
| α | r or 2r | (Unsloth) |
| Learning rate | ~2e-4 (SFT), ~10× the full-FT optimum; ~5e-6 for DPO/GRPO | (LoRA Without Regret, Unsloth); TRL suggests ~1e-4 for SFT adapters and ~1e-5 for DPO adapters (TRL) |
| Dropout | 0–0.1 | Rarely useful on short runs (Unsloth) |
| Batch size | Modest (effective ~16) | LoRA degrades more than full FT at large batch, and more rank doesn’t fix it (LoRA Without Regret) |
Note that the original paper adapted only Wq and Wv with tiny ranks and still matched full fine-tuning on GPT-3 (LoRA). Later work found all-layer LoRA matters at modern scale and data sizes (LoRA Without Regret).
LoRA vs full fine-tuning
| LoRA / QLoRA | Full fine-tuning | |
|---|---|---|
| Memory | Base weights + small adapter + its optimizer state | Weights + gradients + full Adam state |
| Storage and serving | MB-scale adapters; many per base model; hot-swappable | One full copy per task |
| Small/medium SFT, RL | Matches full FT if all layers are adapted (LoRA Without Regret) | Baseline |
| Large-corpus code/math, continued pre-training | Substantially worse (Biderman et al.) | Better; learns perturbations of 10–100× higher rank |
| Forgetting | Less: preserves out-of-domain skills and diversity (Biderman et al.) | More |
Google’s managed tuning shows the same split: adapter tuning is cheaper and works with less data, while full tuning may score higher on complex tasks but costs more to train and serve (Vertex AI).
Failure modes
- Capacity ceiling. On large datasets the adapter can’t absorb everything, and loss plateaus above full fine-tuning (LoRA Without Regret, Biderman et al.).
- Wrong LR. Copying a full-FT learning rate into LoRA undertrains by about 10× (LoRA Without Regret).
- Attention-only targeting. This older default leaves quality on the table (LoRA Without Regret).
- Overfitting at high rank or too many epochs. Training loss falling below ~0.2 is a warning sign (Unsloth).
- Adapter/base mismatch. Synthesis: an adapter only works on the exact base checkpoint (and quantization) it was trained against. Version adapters with their base.
When to use, and when not
- Use LoRA/QLoRA by default for task, format and style SFT, for DPO and RL, for limited GPU memory, for many tenant- or task-specific variants served from one base (LLM inference serving), and when you want to limit forgetting.
- Prefer full fine-tuning for continued pre-training on billions of tokens, for large domain shifts such as a new language or heavy code or math, or when you have the compute and evals show LoRA hitting a ceiling.