Open Athena | Scaling Laws That Extrapolate 300× Past the Fit
Delphi is an open scaling suite ranging from 3e18 to 1e23 FLOPs. A pre-registered forecast from its scaling law predicted the loss of the largest run within 0.2%, extrapolating 300× past the...
In this post, I'll describe the process of developing Delphi, the Marin team's first open scaling suite, inspired by Pythia. Delphi has three parts: a scaling recipe that maps compute budgets to model configurations, a scaling suite trained from that recipe on the Google TPU Research Cloud, and a scaling law that uses the smaller Delphi models to predict the larger ones. We release the checkpoints, training mixture, and recipe so Delphi can serve as a new resource for scaling studies. A pre-registered forecast from the scaling law predicted the final loss of the largest Delphi run (1e23…
saved by
related reading
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- [Hero Run] 535B-A23B on 18T tokensgithub.com
- Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scaleopenathena.ai
- DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- The Scaling Hypothesis · Gwern.netgwern.net
- [2509.14786] Pre-training under infinite computearxiv.org
- Scaling is subtler than it seemsberen.io
- 10x Data Efficiency - NanoGPT Slowrunqlabs.sh
- How To Scale Your Modeljax-ml.github.io
- Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com
- >10x More Efficient Pretraining — Magicmagic.dev