Source: Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

A 2024 Google study showing that SFT examples with unknown facts are learned slowly and raise hallucination.

Contents
  • What: research paper (arXiv:2405.05904, EMNLP 2024)
  • Author: Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig
  • Date: 2024-05-09 (v3 2024-10-01)
  • Original: arXiv:2405.05904

Summary

Controls the share of fine-tuning examples that contain facts the model did not already know, and measures the effect on hallucination.

Key claims

  • Examples introducing new knowledge are learned significantly slower than examples consistent with what the model knows.
  • Once learned, they linearly increase the model’s tendency to hallucinate.
  • LLMs mostly acquire facts in pre-training; fine-tuning mainly teaches them to use what they know.

Feeds

Fine-tuning

Suggest changes →