Source: The Llama 4 herd (Meta AI blog)
Meta's Llama 4 launch: Scout (109B/17B active, 16 experts) and Maverick (400B/17B active, 128 routed + 1 shared), Meta's first MoE models, with alternating dense and MoE layers.
Contents
- What: vendor engineering blog post
- Publisher: Meta AI
- Date: 2025-04-05
- Original: The Llama 4 herd
Summary
Meta’s switch from dense Llama 3 to MoE. Maverick alternates dense and MoE layers; in MoE layers each token goes to a shared expert plus exactly one of 128 routed experts. Meta stresses that all parameters stay in memory even though only a subset is active.
Key claims
- Scout: 17B active, 109B total, 16 experts, 10M-token context; fits one H100 with Int4 quantisation.
- Maverick: 17B active, 400B total, 128 routed + 1 shared expert; fits one H100 DGX host.
- Behemoth (preview): 288B active, 16 experts, nearly 2T total; teacher for co-distilling Maverick; pre-trained in FP8 on 32K GPUs at 390 TFLOPs/GPU.
- Alternating dense and MoE layers is chosen for inference efficiency.