Source: An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning

A 2023 study measuring how continual instruction tuning erodes LLMs' prior knowledge and skills.

Contents
  • What: research paper (arXiv:2308.08747)
  • Author: Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang
  • Date: 2023-08-17 (v5 2025-01-05)
  • Original: arXiv:2308.08747

Summary

Measures forgetting of domain knowledge, reasoning and reading comprehension in 1B–7B models during continual instruction tuning.

Key claims

  • Catastrophic forgetting appears across 1B–7B models.
  • In that range, forgetting got more severe as scale increased.
  • Decoder-only BLOOMZ forgot less than encoder-decoder mT0.
  • General instruction tuning beforehand helps mitigate later forgetting.

Feeds

Fine-tuning

Suggest changes →