MIA: Greg Yang, Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer - YouTube

Models, Inference and AlgorithmsBroad InstituteApril 6, 2022This talk was presented in 2 parts.Meeting: Tuning Large Neural Networks via Zero-Shot Hyperparam...

Models, Inference and Algorithms Broad Institute April 6, 2022 This talk was presented in 2 parts. Meeting: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer Greg Yang Deep Learning Group, Microsoft Research "You can't train GPT-3 on a single GPU, much less tune its hyperparameters (HPs). I'm here to tell you this is not true: you can tune its HPs on a single GPU — even if you can't train it that way! In the first hour of this talk, I'll describe how, in the so-called “maximal update parametrization” (abbreviated µP), narrow and wide neural networks share the same set…

saved by

• Roy Liao

related reading

• Training Dynamics in Large Networks: From Super-Wide to the Scaling Law Regimeyoutube.com
• Greg Yang | Professional pagethegregyang.com
• The Scaling Hypothesis · Gwern.netgwern.net
• The Practitioner’s Guide to the Maximal Update Parameterization - Cerebrascerebras.ai
• How To Scale Your Modeljax-ml.github.io
• The Practitioner's Guide to the Maximal Update Parameterization | EleutherAI Blogblog.eleuther.ai
• A Proof of Learning Rate Transfer under $\mu$Parxiv.org
• Infinite Limits of Neural Networks - Kempner Institutekempnerinstitute.harvard.edu
• Deep Dive into LLMs like ChatGPTyoutube.com
• Let's build GPT: from scratch, in code, spelled out.youtube.com
• Neural Networks: Zero To Herokarpathy.ai
• Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operationsyoutube.com