Deep Neural Nets: 33 years ago and 33 years from now

Musings of a Computer Scientist.

The Yann LeCun et al. (1989) paper Backpropagation Applied to Handwritten Zip Code Recognition is I believe of some historical significance because it is, to my knowledge, the earliest real-world application of a neural net trained end-to-end with backpropagation. Except for the tiny dataset (7291 16x16 grayscale images of digits) and the tiny neural network used (only 1,000 neurons), this paper reads remarkably modern today, 33 years later - it lays out a dataset, describes the neural net architecture, loss function, optimization, and reports the experimental classification error rates over t

related reading

• Deep Neural Nets: 33 years ago and 33 years from nowkarpathy.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• Andrej Karpathy blogkarpathy.github.io
• The Decade of Deep Learning | Leo Gaobmk.sh
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
• Yes you should understand backprop | by Andrej Karpathy | Mediumkarpathy.medium.com
• The Little Book of Deep Learningfleuret.org
• Neural networks: from the perceptron to deep netsarxiv.org
• GitHub - adam-maj/deep-learning: A deep-dive on the entire history of deep-learninggithub.com