IsoFLOP curves of large language models are extremely flat – Severely Theoretical
An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchil…
An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchilla scaling laws (more than two years ago) to argue for training smaller models (point 4 in that post). I’m glad that this observation is finally being taken seriously, but I think the quotation above from the Llama-3 paper still underestimates the extent of this isoFLOP flatness issue. The performance of these models is not just robust to small variations in model size around the optimal, but it…
saved by
related reading
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- Chinchillaarxiv.org
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- New Scaling Laws for Large Language Models — LessWronglesswrong.com
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- [2510.03280] Training Optimal Large Diffusion Language Modelsar5iv.labs.arxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- chinchilla's wild implications — AI Alignment Forumalignmentforum.org
- chinchilla's wild implications — LessWronglesswrong.com
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- Scaling: The State of Play in AIoneusefulthing.org
- Training a compute-optimal gpt2-small – Tomek Korbak — personal homepagetomekkorbak.com