Source: Distilling the Knowledge in a Neural Network
Hinton, Vinyals and Dean's 2015 paper that named and formalised knowledge distillation.
Contents
- What: research paper (arXiv:1503.02531, NIPS 2014 Deep Learning Workshop)
- Author: Geoffrey Hinton, Oriol Vinyals, Jeff Dean (Google)
- Date: 2015-03-09
- Original: arXiv:1503.02531
Summary
Compresses an ensemble or large model into a small one by training the small model on the large model’s temperature-softened class probabilities. Shows results on MNIST and a commercial speech recogniser, and introduces specialist models.
Key claims
- Raise the softmax temperature T so the teacher’s output is soft; use the same T for the student during training and T=1 afterwards.
- Best objective: weighted average of cross-entropy with soft targets (at T) and with true labels (at T=1), weighting the hard term lower.
- Soft-target gradients scale as 1/T², so multiply that term by T².
- At high temperature, distillation reduces to matching logits; for very small students, intermediate T (2.5–4 on MNIST) worked best.
- MNIST: a small net went from 146 errors to 74 when trained on soft targets from a 67-error teacher.
- Soft targets act as a regulariser: a speech model trained on 3% of data generalised well with soft targets but overfit with hard ones.