chinchilla's wild implications - LessWrong

The DeepMind paper that introduced Chinchilla revealed that we've been using way too many parameters and not enough data for large language models. T…

x chinchilla's wild implications — LessWrong Best of LessWrong 2022 Scaling Laws Language Models (LLMs) Machine Learning (ML) DeepMind AI Frontpage 425 chinchilla's wild implications by nostalgebraist 31st Jul 2022 AI Alignment Forum 13 min read 129 425 Ω 97 ( Colab notebook here.) This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla. [1] The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: Data, not size, is the current

related reading

• chinchilla's wild implications — AI Alignment Forumalignmentforum.org
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• Chinchillaarxiv.org
• New Scaling Laws for Large Language Models — LessWronglesswrong.com
• The Scaling Hypothesis · Gwern.netgwern.net
• Will scaling work?dwarkeshpatel.com
• Training a compute-optimal gpt2-small – Tomek Korbak — personal homepagetomekkorbak.com
• IsoFLOP curves of large language models are extremely flatseverelytheoretical.wordpress.com
• Scaling is subtler than it seemsberen.io
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Go smol or go home | Harm de Vriesharmdevries.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org