chinchilla's wild implications — AI Alignment Forum

(Colab notebook here.) • This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla.[1] …

x chinchilla's wild implications — AI Alignment Forum Best of LessWrong 2022 Scaling Laws Language Models (LLMs) Machine Learning (ML) DeepMind AI Frontpage 97 chinchilla's wild implications by nostalgebraist 31st Jul 2022 13 min read 129 97 ( Colab notebook here.) This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla. [1] The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: Data, not size, is the currently active constra

related reading

• chinchilla's wild implications — LessWronglesswrong.com
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• New Scaling Laws for Large Language Models — LessWronglesswrong.com
• Chinchillaarxiv.org
• How To Scale Your Modeljax-ml.github.io
• Lil'Loglilianweng.github.io
• Training a compute-optimal gpt2-small – Tomek Korbak — personal homepagetomekkorbak.com
• The Scaling Hypothesis · Gwern.netgwern.net
• Will scaling work?dwarkeshpatel.com
• IsoFLOP curves of large language models are extremely flatseverelytheoretical.wordpress.com
• [2510.03280] Training Optimal Large Diffusion Language Modelsar5iv.labs.arxiv.org
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com