Source: An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning
A 2023 study measuring how continual instruction tuning erodes LLMs' prior knowledge and skills.
Contents
- What: research paper (arXiv:2308.08747)
- Author: Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, Yue Zhang
- Date: 2023-08-17 (v5 2025-01-05)
- Original: arXiv:2308.08747
Summary
Measures forgetting of domain knowledge, reasoning and reading comprehension in 1B–7B models during continual instruction tuning.
Key claims
- Catastrophic forgetting appears across 1B–7B models.
- In that range, forgetting got more severe as scale increased.
- Decoder-only BLOOMZ forgot less than encoder-decoder mT0.
- General instruction tuning beforehand helps mitigate later forgetting.