✳flâneur — a map of the web's best reading

Understanding Muon

lakernewhouse.com · 2,342 words · saved by 1 readers

Our series here is inspired by all three sources above. In addition, this series is about my master's research this year at MIT. Tracing Muon's scholarly history has many branches, including Shampoo, spectral descent, and duality descent all pre-2020. Jeremy Bernstein picked up with his PhD thesis, spectral norm work, and modular norm work, which led to our optimizer anthology based on Jeremy's ideas. The anthology proposed an efficient Newton-Schulz iteration for setting singular values to 1. The iteration inspired Keller Jordan to try the resulting optimizer on two competitive benchmarks, CIFAR-10 speedrun and NanoGPT speedrun. Since then, Ashish Vaswani and Jianlin Su showed Muon scales to

Understanding Muon Understanding Muon Chapter 1: Into the Matrix Speedy and fun Muon explainer Chapter 2: Source Code Annotated PyTorch implementation Chapter 3: Weight Regulation Our work, and future directions Learn with LLM (what's this?) Understanding Muon Chapter 1: Into the Matrix July 21, 2025 "To understand Muon, the first step is to enter the Matrix." —Not quite Morpheus We have seen a lot of appreciation for the Muon optimizer, and excitement that it's been scaled up to training a 1T parameter model . --> A lot of people are learning about Muon for the first time. The first goal of t

Explore this link on the map →

saved by

related reading