[2304.06636] Neural networks: from the perceptron to deep nets
Abstract:Artificial networks have been studied through the prism of statistical mechanics as disordered systems since the 80s, starting from the simple models of Hopfield's associative memory and the single-neuron perceptron classifier. Assuming data is generated by a teacher model, asymptotic generalisation predictions were originally derived using the replica method and the online learning dynamics has been described in the large system limit. In this chapter, we review the key original ideas of this literature along with their heritage in the ongoing quest to understand the efficiency of modern deep learning algorithms. One goal of current and future research is to characterize the bias of the learning algorithms toward well-generalising minima in a complex overparametrized loss landscapes with many solutions perfectly interpolating the training data. Works on perceptrons, two-layer committee machines and kernel-like learning machines shed light on these benefits of overparametrization. Another goal is to understand the advantage of depth while models now commonly feature tens or hundreds of layers. If replica computations apparently fall short in describing general deep neural networks learning, studies of simplified linear or untrained models, as well as the derivation of scaling laws provide the first elements of answers.
View PDF HTML (experimental) Abstract:Artificial networks have been studied through the prism of statistical mechanics as disordered systems since the 80s, starting from the simple models of Hopfield's associative memory and the single-neuron perceptron classifier. Assuming data is generated by a teacher model, asymptotic generalisation predictions were originally derived using the replica method and the online learning dynamics has been described in the large system limit. In this chapter, we review the key original ideas of this literature along with their heritage in the ongoing quest to…
saved by
related reading
- nn-notes.pdfboris-hanin.github.io
- [2007.12826] The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingarxiv.org
- Deep Learning is Not So Mysterious or Differentarxiv.org
- Neural networks and deep learningneuralnetworksanddeeplearning.com
- [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
- Statistical Mechanics of Deep Learningganguli-gang.stanford.edu
- Understanding deep learning requires rethinking generalizationarxiv.org
- Infinite Limits of Neural Networks - Kempner Institutekempnerinstitute.harvard.edu
- There Will Be a Scientific Theory of Deep Learningarxiv.org
- NL.pdfabehrouz.github.io
- Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
- The Little Book of Deep Learningfleuret.org