Fine-tuning

Continuing to train a pre-trained LLM's weights on task data (SFT, preference or RL) to change how it behaves; when to use it versus prompting or RAG, how to prepare data, and how it fails.

Contents

Fine-tuning means continuing to train a pre-trained language model’s weights on a smaller, targeted dataset so that it behaves differently: it follows a format, adopts a style, gets better at a narrow task, or makes different choices. Compare the cheaper alternatives. Prompting changes how you ask. RAG changes what the model can see at query time. Fine-tuning changes how the model behaves, and it costs the most effort of the three (notebook). Post-training has three families: supervised fine-tuning (SFT), preference optimization (RLHF, DPO, GRPO), and distillation. Each can update all weights or only a small adapter (PEFT/LoRA).

Fine-tune, prompt, or RAG?

PromptingRAGFine-tuning
ChangesHow you askWhat it knows nowHow it behaves
EffortLowestMedium (retrieval infra)Highest (data, training, evals)
Freshnessn/aLive: update the indexStale: retrain to update
IterationInstantFastSlow
Best forMost tasks; try firstPrivate or changing facts, citationsDurable format, style, niche skill; shrinking long prompts

Source: notebook. The vendors give the same order. OpenAI says to write evals first, then prompt, and to fine-tune only if the use case calls for it (OpenAI). Google says to start with prompting and to analyse failures before adding data (Vertex AI).

Fine-tuning is the wrong tool for teaching facts. The research supports this:

  • RAG consistently beat unsupervised fine-tuning (continued training on documents) for knowledge injection, for both familiar and entirely new facts (Ovadia et al.).
  • SFT examples containing facts the model didn’t already know are learned more slowly. Once learned, they linearly increase hallucination. Fine-tuning mostly teaches a model to use knowledge it got in pre-training (Gekhman et al.). See hallucination.
  • LIMA aligned a 65B model with only 1,000 curated SFT examples. This supports the view that alignment mostly teaches format and style on top of knowledge from pre-training (LIMA).

Fine-tuning does pay off in these cases (OpenAI, Vertex AI):

  • strict output formats (structured outputs), classification and extraction
  • a consistent tone
  • fixing instruction-following failures
  • more examples than fit in a prompt
  • replacing a long few-shot prompt, which cuts per-call cost and latency
  • a smaller, cheaper model for one task (often by distillation)

Mature systems combine all three: a fine-tuned model for behaviour, retrieved facts for grounding, and a well-built prompt (notebook).

flowchart TD
  A["Need to adapt a model"] --> B["Write evals and a baseline"]
  B --> C["Prompt engineering"]
  C -->|"good enough"| Z["Ship"]
  C -->|"missing private or fresh facts"| D["Add RAG"]
  D -->|"good enough"| Z
  C -->|"behaviour, format or style won't hold"| E["Fine-tune: SFT"]
  D -->|"behaviour still off"| E
  E -->|"need preferences or verifiable skill"| F["DPO or RL (GRPO / RFT)"]
  E --> G["Eval vs baseline and general-ability regression"]
  F --> G
  G -->|"pass"| Z
  G -->|"fail"| H["Fix data, not just hyperparameters"]
  H --> E

Synthesis: the diagram combines the notebook’s decision order with the vendor eval loops.

The methods

  • SFT (supervised fine-tuning). Token-level cross-entropy (negative log-likelihood) on target sequences. Padding is always masked. For prompt/completion data TRL also masks the prompt by default, so loss falls only on the completion; for conversational data, masking everything but assistant turns is opt-in (assistant_only_loss=True) (TRL). InstructGPT’s first stage was SFT on about 13k labeller demonstrations (InstructGPT).
  • Preference tuning (DPO and similar). Trains on chosen/rejected pairs. Suits subjective qualities such as tone or summary focus. See preference optimization.
  • Reinforcement fine-tuning (RFT/RLFT). Reward functions score sampled outputs. It suits verifiable, multi-step tasks. Vertex AI says it can need only dozens to a few hundred prompts (Vertex AI); Bedrock scores responses with reward functions such as AWS Lambda (Bedrock). See reinforcement fine-tuning.
  • Continued pre-training. Next-token training on raw domain text. It forgets the most and is the weakest way to add facts (LoRA Learns Less, Ovadia et al.).

Full fine-tuning vs PEFT

Full fine-tuning updates every parameter. You need memory for the weights, the gradients and the Adam optimizer states. On GPT-3 175B that meant 1.2 TB of training VRAM, against 350 GB with LoRA (LoRA). PEFT freezes the base model and trains a small adapter. QLoRA goes further and freezes the base in 4-bit (quantization), which fits a 65B model on one 48 GB GPU (QLoRA).

The trade-offs: LoRA “learns less and forgets less”. It underperforms full fine-tuning on large code and math corpora but keeps more of the base model’s abilities (Biderman et al.). On small-to-medium SFT sets and on RL, LoRA applied to all layers matches full fine-tuning (LoRA Without Regret). Vertex AI offers both: parameter-efficient tuning is cheaper and needs less data, while full tuning may score higher but costs more to train and serve (Vertex AI). Details are on PEFT and LoRA.

Data preparation

Data quality matters more than any hyperparameter:

  • Quality over quantity. LIMA got there with 1,000 curated examples (LIMA). QLoRA trained over 1,000 models and found that a small, high-quality dataset beat larger ones (QLoRA). Vertex AI recommends hundreds to thousands of well-labelled examples for SFT and puts quality ahead of volume (Vertex AI).
  • Formats. TRL accepts plain text, prompt/completion, or conversational messages (role/content). It applies the model’s chat template for you. Preference data is prompt/chosen/rejected (TRL). Vendor APIs take JSONL (OpenAI).
  • Chat template and EOS. Train with the same template you’ll serve with. A mismatched EOS token gives a model that never stops. For example, TRL notes that Qwen2.5 base needs eos_token="<|im_end|>" (TRL).
  • Loss masking. Train on completions or assistant turns only. Unsloth reports a small accuracy gain, larger in multi-turn data (Unsloth).
  • Packing. Concatenating short examples into fixed-length blocks cuts padding waste (TRL packing=True, Axolotl multipacking) (TRL, Axolotl).
  • Avoid unknown facts in SFT targets. They raise hallucination (Gekhman et al.). Synthesis: filter or rewrite answers whose facts the base model can’t support, and put those facts in RAG instead.
  • Hold-out evals on representative real data, plus general-capability regression tests (OpenAI). See LLM evaluation.

Key hyperparameters (starting points)

KnobTypical startNotes
Learning rate2e-5 full SFT (TRL default); 1e-4 to 2e-4 LoRA; 1e-6 to 5e-6 DPO/GRPOLoRA’s optimal LR is about 10× full fine-tuning’s
Epochs1–3More epochs overfit instruction data
Effective batch~16LoRA is hurt more by very large batches
Max length1024 (TRL default)Truncation can cut off answers or image tokens

Sources: TRL, Unsloth, LoRA Without Regret. Unsloth treats a training loss below ~0.2 as a sign of overfitting (Unsloth).

Failure modes

  • Catastrophic forgetting. Narrow fine-tuning erodes general skills. In 1B–7B models, continual instruction tuning eroded domain knowledge, reasoning and reading comprehension, and the damage grew with scale in that range. Prior general instruction tuning reduced it (Luo et al.). A concrete case: mid-training Qwen3-8B on internal documents dropped IF-eval from 85% to 45%. Mixing 30% chat data only partly helped (79%). On-policy distillation of that mixed checkpoint, with the original Qwen3-8B as teacher, restored it to 83% while keeping most of the new knowledge (Thinking Machines). Mitigations: LoRA, which forgets less (Biderman et al.); mixing in general data; mixing pre-training gradients, as PPO-ptx does against the RLHF “alignment tax” (InstructGPT); a distillation recovery pass.
  • Hallucination from new facts (Gekhman et al.).
  • Overfitting. The model memorises templates and loses diversity. Fix it with fewer epochs, a lower LR, more varied data or early stopping (Unsloth).
  • Staleness and lock-in. Facts baked into the weights go stale (notebook). A tuned model is tied to its base model’s lifecycle. OpenAI’s fine-tuned models stop working when their base model is retired (OpenAI deprecations).
  • Safety regression. Synthesis: fine-tuning can wear away refusal behaviour learned in the base model’s preference tuning. Re-run safety evals after every tune (see guardrail tooling).

Managed fine-tuning APIs (as of 2026-11-14)

VendorMethodsTunable models (examples)Notes
OpenAISFT, DPO, vision SFTgpt-4.1 / 4.1-mini (SFT, DPO); gpt-4o (vision). gpt-4.1-nano is still listed in the guide but its base model shut down on 2026-10-23Winding down. Closed to orgs that never fine-tuned since 2026-05-07, and to orgs with no fine-tuned-model inference in the past 60 days since 2026-07-02; existing customers can’t create jobs from 2027-01-06. gpt-4.1, gpt-4.1-mini, gpt-4o and gpt-4o-mini (and their fine-tunes) shut down 2027-01-14 (guide, deprecations)
Google Vertex AISFT, preference tuning, RLFT; checkpoints and continuous tuningGemini 3.5 Flash/Flash-Lite, 3.1 Flash-Lite, 2.5 Pro/Flash/Flash-Lite (varies by method)PEFT or full (Vertex AI)
Amazon BedrockSFT, reinforcement fine-tuning (Lambda rewards), distillationNova family, Claude 3 Haiku, Llama 3.xBilled per training token × epochs, plus monthly storage (Bedrock)

Synthesis: the managed path is shrinking at OpenAI. Open-weight fine-tuning (open-weight models) with TRL, Unsloth or Axolotl gives you portability and control over the model’s lifecycle, but you own serving it (LLM inference serving).

Open-source tooling

  • Hugging Face TRL. The reference post-training library: SFTTrainer, DPOTrainer, GRPOTrainer, RewardTrainer, KTOTrainer, distillation trainers, PEFT and vLLM integration (TRL). See Hugging Face ecosystem.
  • Unsloth. Single-node speed and memory. It claims about 2× faster training and up to 70% less VRAM, and sells itself on QLoRA and GRPO on consumer GPUs. It exports to GGUF for local LLMs (Unsloth).
  • Axolotl. A whole pipeline in one YAML file. Supports full, LoRA, QLoRA, DPO/KTO/ORPO and GRPO, and scales across GPUs with FSDP2 and DeepSpeed (Axolotl).

Synthesis: pick Unsloth for fast single-GPU experiments, Axolotl for reproducible multi-GPU runs driven by config, and TRL when you need to customise the algorithm or are using it underneath the other two anyway.

When to use, and when not

  • Use it when prompting and RAG plateau on a behavioural gap you can measure, you have at least a few hundred high-quality examples, and you have an eval harness. It is also the right move when cost or latency at scale justifies moving a long prompt into the weights, or distilling into a smaller model (cost and latency optimization).
  • Don’t use it to add or update facts (use RAG), to cover poor prompt design, without evals, or when the base model will be retired before you recoup the effort.

Sources

Suggest changes →