Source: Will We Run Out of Data? (Villalobos et al., Epoch AI, 2024)
Epoch AI projection that frontier training sets will match the whole stock of public human text between 2026 and 2032, sooner if models are over-trained.
Contents
- What: research paper (arXiv:2211.04325), ICML 2024 position paper
- Authors: Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn
- Date: 2022-10-26 (v2 2024-06-04)
- Original: arxiv.org/abs/2211.04325
Summary
Estimates the stock of publicly available human-written text and projects dataset growth for frontier models. If trends continue, training sets reach the size of that stock within a few years, making data rather than compute the binding constraint.
Key claims
- Frontier models would be trained on datasets about the size of the public human text stock between 2026 and 2032.
- Over-training (more tokens per parameter than compute-optimal) moves that date earlier.
- Synthetic data, transfer from other modalities and better data efficiency are discussed as ways to keep scaling.