The Practitioner's Guide to the Maximal Update Parameterization | EleutherAI Blog

Exploring the implementation details of muTransfer

Table of Contents Introduction Why you should use μP 1. Stable optimum HPs across scale (μTransfer) 2. Improved loss at large scale due to improved HP tuning 3. Stable training - significantly decreased danger of instability at large scale 4. More predictable scaling due to μTransfer μP enables better research A Simple Approach to the μP Math Basic Building Block: Controlled Activation Magnitudes Operations in a training step Practitioner's guide to μP Implementation Coordinate check test μTransfer test Transferring optimal HPs from a small scale to a large scale Conclusion Citation Footnotes

saved by

• Paul Scotti
• Akira Yoshiyama
• Ishan Mukherjee

related reading

• The Practitioner’s Guide to the Maximal Update Parameterization - Cerebrascerebras.ai
• Deriving Muonjeremybernste.in
• Does Muon improve regulatory DNA learning? Part 1. — Origin Bioorigin.bio
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Marketplace: my first attempt at training without backprop on GPU efficiently – Fang-Pen's coding notefangpenlin.com
• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
• A Proof of Learning Rate Transfer under $\mu$Parxiv.org
• Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
• microgptkarpathy.github.io
• Scaling is subtler than it seemsberen.io
• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai