A Recipe for Training Neural Networks

Musings of a Computer Scientist.

Some few weeks ago I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them

saved by

• Brandon Wang
• Julian Quevedo
• Saeejith Nair
• Ivy Zhang
• CLL
• Vincent Huang

related reading

• A Recipe for Training Neural Networkskarpathy.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Neural network training makes beautiful fractals | Jascha’s blogsohl-dickstein.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• Feature Visualizationdistill.pub
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• The Little Book of Deep Learningfleuret.org
• Deep Neural Nets: 33 years ago and 33 years from nowkarpathy.github.io
• Deep Neural Nets: 33 years ago and 33 years from nowkarpathy.github.io
• Bayesian Neural Networkscs.toronto.edu
• The Decade of Deep Learning | Leo Gaobmk.sh