A Recipe for Training Neural Networks

Musings of a Computer Scientist.

Some few weeks ago I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar :)). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them

saved by

• Winnie Xu
• Claire Wang
• Ivy Zhang
• Jordan Sucher
• Uzay Girit
• Andrew Kachnic
• Kenson Hui
• Jacob G-W
• Gloria Ma
• Tomi Jaga
• Liam Hinzman
• Curtis Chong

related reading

• A Recipe for Training Neural Networkskarpathy.github.io
• Debugging Reinforcement Learning Systemsandyljones.com
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Making Deep Learning go Brrrr From First Principleshorace.io
• Bayesian Neural Networkscs.toronto.edu
• Just Ask for Generalization | Eric Jangevjang.com
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• [2403.04807] Mathematics of Neural Networks (Lecture Notes Graduate Course)arxiv.org
• Feature Visualizationdistill.pub