Source: 2 OLMo 2 Furious (Team OLMo, Ai2, 2025)

Ai2's fully open OLMo 2 (7B/13B/32B) report, documenting two-stage pretraining with a mid-training anneal on Dolmino Mix 1124, model souping and Tulu 3-style post-training.

Contents
  • What: technical report (arXiv:2501.00656); short version at COLM 2025
  • Authors: Team OLMo (Pete Walsh, Luca Soldaini, Dirk Groeneveld, … Noah A. Smith, Hannaneh Hajishirzi), Allen Institute for AI
  • Date: 2024-12-31 (v3 2025-10-08)
  • Original: arxiv.org/abs/2501.00656

Summary

Releases weights, data, code, logs and intermediate checkpoints for dense 7B, 13B and 32B models. Pretraining is split into a long web-heavy stage and a short “mid-training” stage in which the learning rate anneals to zero on curated and synthetic data, and the resulting checkpoints are averaged. Post-training follows Tulu 3 (SFT → DPO → RLVR).

Key claims

  • Stage 1 (about 90-95% of FLOPs): OLMo 2 Mix 1124, about 3.9T tokens, over 95% web (DCLM-Baseline 3.71T) plus code, papers and maths. Total tokens: 7B about 4T, 13B about 5T, 32B about 6T.
  • Stage 2, mid-training (about 5-10% of FLOPs): Dolmino Mix 1124, with filtered high-quality web, FLAN, peS2o, Wikipedia, Stack Exchange, and about 10.7B tokens of mostly synthetic maths. Budgets of 50B, 100B or 300B tokens, LR decayed linearly to zero.
  • Several mid-training runs with different data orders are averaged (“souped”), which consistently beats single checkpoints.
  • Mid-training raised the dev average by about 10 points; GSM8K went from 24.1 to 67.5 (7B) and 37.3 to 75.1 (13B).
  • The term “mid-training” is credited to Microsoft’s Phi work and OpenAI.
  • Post-training: SFT, on-policy preference tuning with DPO, then multi-stage RLVR.

Feeds

LLM training pipeline

Suggest changes →