✳flâneur — a map of the web's best reading

Deriving Muon

jeremybernste.in · 3,276 words · saved by 15 readers

We recently proposed Muon: a new neural net optimizer. Muon has garnered attention for its excellent practical performance: it was used to set NanoGPT speed records leading to interest from the big labs. What makes Muon particularly special to me is that we derived the core numerical methods from an exact theoretical principle. This is in contrast to popular optimizers like Adam, which have more heuristic origins and often converge slower than Muon. In this post, I will walk through a derivation of Muon. I hope this will provide context that may help researchers extend the methods to new layer types and beyond. While this post focuses on the theory behind Muon, I recommend checking out Keller’s post to learn more about the algorithm—including the substantial ingenuity that went into making the implementation run fast. Muon is an optimizer specifically designed for Linear neural network layers. By Linear , I mean layers that take an input vector 𝑥 and multiply by a weight matrix

Deriving Muon jeremybernste.in research teaching writing cv Deriving Muon Boston, 7 Mar 2025 Particle tracks in a bubble chamber. Fermilab. We recently proposed Muon : a new neural net optimizer. Muon has garnered attention for its excellent practical performance: it was used to set NanoGPT speed records leading to interest from the big labs . What makes Muon particularly special to me is that we derived the core numerical methods from an exact theoretical principle. This is in contrast to popular optimizers like Adam , which have more heuristic origins and often converge slower than Muon . In

Explore this link on the map →

saved by

related reading