>10x More Efficient Pretraining — Magic

Research update on compute-efficient pretraining and scaling to trillion-parameter models.

Research update on compute-efficient pretraining and scaling to trillion-parameter models. Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. After compounding for … a while …, our pretraining recipe is now >10x more compute-efficient than that of leading open-weight base models. We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200. We continued scaling 10x (~$4M) and meaningfully outperformed all publicly available open base models on…

saved by

• Anton Otaner
• Nathan Chen

related reading

• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Pretraining progress is mostly coming from datadwarkesh.com
• [2509.14786] Pre-training under infinite computearxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• AI progress is about to speed up | Epoch AIepoch.ai
• Training Imperativesdan.io
• 10x Data Efficiency - NanoGPT Slowrunqlabs.sh
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• Pre-training under infinite computearxiv.org
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org