How we can make machine learning algorithms tunable
In the previous blogpost, we established that machine learning algorithms are often hard to tune, and hopefully explained the mechanism for why gradient descent has difficulty with linear losses. In this blogpost, we will lay out some possible solutions.
30th January 2021 How we can make machine learning algorithms tunable Jonas Degrave , Ira Korshunova How we can make machine learning algorithms tunable Jonas Degrave , Ira Korshunova Machine Learning 10 min read In the previous blog post, we established that machine learning algorithms are often hard to tune , and hopefully explained the mechanism for why gradient descent has difficulty with linear combinations of losses. In this blog post, we will lay out some possible solutions. But first, let us identify what we mean by "tunable algorithm". What is it that makes a hyper-parameter easy to t
related reading
- Hyperparameter optimization - Wikipediaen.wikipedia.org
- Modular Manifolds - Thinking Machines Labthinkingmachines.ai
- Why Momentum Really Worksdistill.pub
- An overview of gradient descent optimization algorithmsruder.io
- Keep the gradient flowingfa.bianp.net
- Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
- [1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systemsarxiv.org
- GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
- Effortless optimization through gradient flows – Machine Learning Research Blogfrancisbach.com
- The Continual Learning Problemjessylin.com
- Scaling laws of optimization – Machine Learning Research Blogfrancisbach.com
- Highly optimized optimizers - by Ben Recht - arg minargmin.net