Scaling Laws Literature Review

I have collected a database of scaling laws for different tasks and architectures, and reviewed dozens of papers in the scaling law literature.

Scaling laws literature review | Epoch AI Common shape of a scaling law, taken from Hestness et al. (2017) Executive summary Scaling laws are predictable relations between the scale of a mode and performance or other useful properties. I have collected a database of scaling laws for different tasks and architectures, and reviewed dozens of papers in the scaling law literature. My main takeaways are: Functional forms : a basic power law can effectively model the scaling behavior in the power-law region but not the transitions to the other two regions. For this, either the M4 estimator or the BN

related reading

• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• Scaling is subtler than it seemsberen.io
• The Scaling Hypothesis · Gwern.netgwern.net
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• How To Scale Your Modeljax-ml.github.io
• Scaling Laws Do Not Scalearxiv.org
• Scaling Laws for Value-Based RLvalue-scaling.github.io
• Lil'Loglilianweng.github.io
• Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
• Scaling laws of optimization – Machine Learning Research Blogfrancisbach.com