✳flâneur — a map of the web's best reading
Modular Manifolds - Thinking Machines Lab
thinkingmachines.ai · 4,437 words · saved by 15 readers
A geometric framework for co-designing neural net optimizers with manifold constraints.
When we train large neural networks, we need to keep them healthy. We do not want the tensors in the network-either the weights, activations or gradients-to grow too large or too small. Very small and very large tensors cause a variety of problems not just limited to numerical underflow and overflow. For example, weight matrices changing size during training makes it harder to design training algorithms-since the relative size of updates to weights has a significant impact on the speed of learning. The gold standard for keeping tensors healthy is to normalize them. Normalization is commonplace
Explore this link on the map →saved by
- Elizabeth Qiu
- Rishi Kothari
- Asher P
- Sudarsh K
- Yudhister Joel Kumar
- Timothy Kostolansky
- Sarah
- Donald
- Divit Rawal
- Idhant Gulati
- Alexiy Buynitsky
- Sam B
related reading
- Deriving Muonjeremybernste.in
- Understanding Muonlakernewhouse.com
- Why Momentum Really Worksdistill.pub
- [2410.21265] Modular Duality in Deep Learningarxiv.org
- Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
- Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
- Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
- The World Inside Neural Networksgoodfire.ai
- Neural network training makes beautiful fractals | Jascha’s blogsohl-dickstein.github.io
- Does Muon improve regulatory DNA learning? Part 1. — Origin Bioorigin.bio
- NL.pdfabehrouz.github.io
- arxiv.org/pdf/2512.24880#page=3.56arxiv.org