[2505.21829] In Search of Adam's Secret Sauce
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack
[2505.21829] In Search of Adam's Secret Sauce Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2505.21829 (cs) [Submitted on 27 May 2025 ( v1 ), last revised 1 Dec 2025 (this version, v2)] Title: In Search of Adam's Secret Sauce Authors: Antonio Orvieto , Robert M. Gower View a PDF of the paper titled In Search of Adam's Secret Sauce, by Antonio Orvieto and 1 other authors View PDF HTML (experimental) Abstract: Understanding the remarkable efficacy of Adam whe
Explore this link on the map →related reading
- Deriving Muonjeremybernste.in
- Does Muon improve regulatory DNA learning? Part 1. — Origin Bioorigin.bio
- Adam Optimizer Causes Privileged Basis in Transformer LM Residual Stream — LessWronglesswrong.com
- Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
- The Annotated Transformernlp.seas.harvard.edu
- Transformer Circuits Threadtransformer-circuits.pub
- Online KL Shampoo | Tildeblog.tilderesearch.com
- Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
- A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
- Why Momentum Really Worksdistill.pub
- An overview of gradient descent optimization algorithmsruder.io
- AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu