A Recipe for Training Neural Networks

Musings of a Computer Scientist.

Some few weeks ago I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them

saved by

• Brandon Wang
• Julian Quevedo
• Saeejith Nair
• Ivy Zhang
• CLL
• Vincent Huang

related reading

• A Recipe for Training Neural Networkskarpathy.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Neural network training makes beautiful fractals | Jascha’s blogsohl-dickstein.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• CS231n Deep Learning for Computer Visioncs231n.github.io
• Game Emulation via Neural Networkmadebyoll.in
• Feature Visualizationdistill.pub
• The Little Book of Deep Learningfleuret.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Bayesian Neural Networkscs.toronto.edu
• Deep Neural Nets: 33 years ago and 33 years from nowkarpathy.github.io