1.1 - Fermi estimate of future training runs

Thanks to Paul Christiano, David Roodman, Joe Carlsmith, and Jared Kaplan for feedback on early drafts of this page, and to Miles Brundage for corrections.

Fermi estimate of future training runs Global risk from deep learning [ List of all pages ] About this site 1 - The case for risk 1.1 - Fermi estimate of future training runs # Budgets and total FLOP of future training runs # Model size via scaling laws # Upper bounds for brain-equivalent capabilities # Training data requirements 1.2 - Applications of high-capability models Coming next - Outline of a research program Get updates via mailing list Last update: 2 August 2022 Daniel Dewey Thanks to Paul Christiano, David Roodman, Joe Carlsmith, and Jared Kaplan for feedback on early drafts of this

saved by

• Alex K. Chen
• Yudhister Joel Kumar
• Danish

related reading

• The Scaling Hypothesis · Gwern.netgwern.net
• Estimating training compute of deep learning models | Epoch AIepochai.org
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• Can AI scaling continue through 2030? | Epoch AIepoch.ai
• Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com
• On neural scaling and the quanta hypothesisericjmichaud.com
• The Little Book of Deep Learningfleuret.org
• Why didn't we get GPT-2 in 2005?dynomight.net
• Scaling: The State of Play in AIoneusefulthing.org
• Chinchillaarxiv.org
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• New Scaling Laws for Large Language Models — LessWronglesswrong.com