How we can make machine learning algorithms tunable
In the previous blogpost, we established that machine learning algorithms are often hard to tune, and hopefully explained the mechanism for why gradient descent has difficulty with linear losses. In this blogpost, we will lay out some possible solutions.
30th January 2021 How we can make machine learning algorithms tunable Jonas Degrave , Ira Korshunova How we can make machine learning algorithms tunable Jonas Degrave , Ira Korshunova Machine Learning 10 min read In the previous blog post, we established that machine learning algorithms are often hard to tune , and hopefully explained the mechanism for why gradient descent has difficulty with linear combinations of losses. In this blog post, we will lay out some possible solutions. But first, let us identify what we mean by "tunable algorithm". What is it that makes a hyper-parameter easy to t
Explore this link on the map →related reading
- Modular Manifolds - Thinking Machines Labthinkingmachines.ai
- Hyperparameter optimization - Wikipediaen.wikipedia.org
- Why Momentum Really Worksdistill.pub
- Deriving Muonjeremybernste.in
- A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
- The Little Book of Deep Learningfleuret.org
- Gradient descent - Wikipediaen.wikipedia.org
- Linear regression: Gradient descent | Machine Learning | Google for Developersdevelopers.google.com
- AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
- Policy Gradient Algorithms | Lil'Loglilianweng.github.io
- Stochastic gradient descent - Wikipediaen.m.wikipedia.org
- Gradient Descent With Momentum from Scratch - MachineLearningMastery.commachinelearningmastery.com