>10x More Efficient Pretraining — Magic

Research update on compute-efficient pretraining and scaling to trillion-parameter models.

Research update on compute-efficient pretraining and scaling to trillion-parameter models. Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. After compounding for … a while …, our pretraining recipe is now >10x more compute-efficient than that of leading open-weight base models. We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200. We continued scaling 10x (~$4M) and meaningfully outperformed all publicly available open base models on…

saved by

• Anton Otaner
• Nathan Chen

related reading

• Composer2.pdfcursor.com
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• [2509.14786] Pre-training under infinite computearxiv.org
• The Scaling Hypothesis · Gwern.netgwern.net
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2509.14786] Pre-training under infinite computearxiv.org
• AI progress is about to speed up | Epoch AIepoch.ai
• Pretraining progress is mostly coming from datadwarkesh.com
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• My picture of the present in AI — LessWronglesswrong.com
• Pre-training under infinite computearxiv.org
• Compression and Intelligencegreene.sh