Source: Mixture of Experts Explained (Hugging Face blog)
Hugging Face's survey of MoE history, load balancing, capacity factor, fine-tuning behaviour and serving trade-offs, written around the Mixtral release.
Contents
- What: explainer blog post
- Authors: Omar Sanseviero, Lewis Tunstall, Philipp Schmid, Sourab Mangrulkar, Younes Belkada, Pedro Cuenca (Hugging Face)
- Date: 2023-12-11
- Original: Mixture of Experts Explained
Summary
A practitioner-oriented review of MoE research from the 1991 “Adaptive Mixture of Local Experts” paper through Shazeer’s sparse gating, GShard, Switch and Mixtral. Strongest on the practical consequences: why routers collapse without balancing, how capacity works, why MoEs fine-tune differently, and when sparse beats dense.
Key claims
- Without intervention the gate favours a few experts, which then get trained more and become more favoured; an auxiliary loss counters this.
- Capacity factor trades quality against communication; Switch did well at 1-1.25. Overflow tokens are passed on via the residual or dropped.
- MoEs overfit more in fine-tuning; higher regularisation inside experts helps.
- Freezing all non-expert weights hurts badly; freezing only the MoE layers is nearly as good as full fine-tuning and faster.
- Instruction tuning benefits MoEs more than dense models (Flan-MoE vs Flan-T5).
- Distilling back to dense keeps 30-40% of the sparsity gains (Switch experiments).
- Sparse MoEs suit high-throughput serving across many machines; dense models suit low-throughput, VRAM-limited settings.
- Mixtral 8x7B needs VRAM for a ~47B dense model but costs about as much per token as a ~12B one.