The Practitioner’s Guide to the Maximal Update Parameterization - Cerebras

Cerebras is the go-to platform for fast and effortless AI training. Learn more at cerebras.ai.

The Practitioner’s Guide to the Maximal Update Parameterization - Cerebras Skip to main content Sep 23 2024 The Practitioner’s Guide to the Maximal Update Parameterization - Cerebras Nolan Dey Quentin Anthony Joel Hestness Introduction Maximal Update Parameterization (µP) offers significant advantages for neural network training, but its adoption has been limited due to the complexity of the underlying math and the challenges in implementation. This guide aims to lower those barriers by providing a clear and practical overview of µP. By using µP, you can achieve stable hyperparameters across m

saved by

• Ishan Mukherjee
• Ali Ahmad

related reading

• The Practitioner's Guide to the Maximal Update Parameterization | EleutherAI Blogblog.eleuther.ai
• A Proof of Learning Rate Transfer under $\mu$Parxiv.org
• How To Scale Your Modeljax-ml.github.io
• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• Deriving Muonjeremybernste.in
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• microgptkarpathy.github.io
• Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
• MIA: Greg Yang, Tuning Large Neural Networks via Zero-Shot Hyperparameter Transferyoutube.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2601.08393] Controlled LLM Training on Spectral Spherearxiv.org