SGD Optimization Algorithms
Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimization algorithms such as Momentum, Adagrad, and Adam actually work. This post explores how many of the most popular gradient-based optimization algorithms actually work. Note: If you are looking for a review paper, this blog post is also available as an article on arXiv. Update 20.03.2020: Added a note on recent optimizers. Update 09.02.2018: Added AMSGrad. Update 24.11.2017: Most of the content in this article is now also available as slides. Update 15.06.2017: Added derivations of AdaMax and Nadam. Update 21.06.16: This post was posted to Hacker News. The discussion provides some interesting pointers to related work and other techniques. Table of contents: Gradient descent is one of the most popular algorithms to perform optimization and by far the most common way to optimiz
This post explores how many of the most popular gradient-based optimization algorithms actually work. Note: If you are looking for a review paper, this blog post is also available as an article on arXiv . Update 20.03.2020: Added a note on recent optimizers . Update 09.02.2018: Added AMSGrad . Update 24.11.2017: Most of the content in this article is now also available as slides . Update 15.06.2017: Added derivations of AdaMax and Nadam . Update 21.06.16: This post was posted to Hacker News. The discussion provides some interesting pointers to related work and other techniques. Table of conten
Explore this link on the map →related reading
- A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
- AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
- Stochastic gradient descent - Wikipediaen.m.wikipedia.org
- Understanding Deep Learning Optimizers: Momentum, AdaGrad, RMSProp & Adam | Towards Data Sciencetowardsdatascience.com
- Why Momentum Really Worksdistill.pub
- Deriving Muonjeremybernste.in
- Modular Manifolds - Thinking Machines Labthinkingmachines.ai
- CS231n Deep Learning for Computer Visioncs231n.github.io
- Gradient descent - Wikipediaen.wikipedia.org
- NL.pdfabehrouz.github.io
- Does Muon improve regulatory DNA learning? Part 1. — Origin Bioorigin.bio
- The Little Book of Deep Learningfleuret.org