Scaling is subtler than it seems

Epistemic status: A lot of this is vibes and generally trying to express tacit knowledge from ML experience and training models. However there are a huge number of details here and the field of ML changes incredibly rapidly. This is an oversimplified, but hopefully still useful picture, and may well...

Epistemic status: A lot of this is vibes and generally trying to express tacit knowledge from ML experience and training models. However there are a huge number of details here and the field of ML changes incredibly rapidly. This is an oversimplified, but hopefully still useful picture, and may well be wrong in some details since in a good number of cases we understand things at a practitioner’s level but not super well theoretically. People nowadays often think of scaling laws as obvious and almost trivial – if you have a bigger model and use more compute to train it, then it will get…

saved by

• Vincent Cheng
• Dhruv Sheth
• Ishan Mukherjee
• Sam B
• Ishaan Panigrahi
• Miguel Villa Floran

related reading

• How To Scale Your Modeljax-ml.github.io
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• A short note on interpretability and mindsericjmichaud.com
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• The Scaling Hypothesis · Gwern.netgwern.net
• Will scaling work?dwarkeshpatel.com
• Pretraining progress is mostly coming from datadwarkesh.com
• On neural scaling and the quanta hypothesisericjmichaud.com
• Scaling: The State of Play in AIoneusefulthing.org