Source: Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
A 2023 Microsoft study finding RAG beats unsupervised fine-tuning for injecting knowledge.
Contents
- What: research paper (arXiv:2312.05934)
- Author: Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha (Microsoft)
- Date: 2023-12-10 (v3 2024-01-30)
- Original: arXiv:2312.05934
Summary
Compares unsupervised fine-tuning (continued training on documents) with RAG on knowledge-intensive tasks across several topics.
Key claims
- RAG consistently outperformed unsupervised fine-tuning, for both previously seen and entirely new facts.
- LLMs struggle to learn new facts via unsupervised fine-tuning.
- Training on many paraphrases of the same fact may partly help.