Modular Manifolds - Thinking Machines Lab
thinkingmachines.ai · 4,437 words · saved by 16 readers
A geometric framework for co-designing neural net optimizers with manifold constraints.
When we train large neural networks, we need to keep them healthy. We do not want the tensors in the network-either the weights, activations or gradients-to grow too large or too small. Very small and very large tensors cause a variety of problems not just limited to numerical underflow and overflow. For example, weight matrices changing size during training makes it harder to design training algorithms-since the relative size of updates to weights has a significant impact on the speed of learning. The gold standard for keeping tensors healthy is to normalize them. Normalization is commonplace
saved by
- Elizabeth Qiu
- Rishi Kothari
- Asher P
- Sudarsh K
- Yudhister Joel Kumar
- Timothy Kostolansky
- Sarah
- Donald
- Divit Rawal
- Idhant Gulati
- Alexiy Buynitsky
- Sam B
related reading
- Deriving Muonjeremybernste.in
- Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
- Understanding Muonlakernewhouse.com
- Ji-Ha's Blogjiha-kim.github.io
- [2604.04891] Muon Dynamics as a Spectral Wasserstein Flowarxiv.org
- Why Momentum Really Worksdistill.pub
- [2410.21265] Modular Duality in Deep Learningarxiv.org
- Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
- Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
- Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
- Single Token Geometry 02: Manifold Tearingsubstack.com
- [1605.08101] Global rates of convergence for nonconvex optimization on manifoldsarxiv.org