Statistical Mechanics of Deep Learning | Annual Review of Condensed Matter Physics

Vol. 11:501-528 (Volume publication date March 2020) First published as a Review in Advance on December 9, 2019 https://doi.org/10.1146/annurev-conmatphys-031119-050745 Yasaman Bahri,1 Jonathan Kadmon,2 Jeffrey Pennington,1 Sam S. Schoenholz,1 Jascha Sohl-Dickstein,1 and Surya Ganguli1,2 1Google Brain, Google Inc., Mountain View, California 94043, USA 2Department of Applied Physics, Stanford University, Stanford, California 94035, USA; email: sganguli@stanford.edu The recent striking success of deep neural networks in machine learning raises profound questions about the theoretical principles underlying their success. For example, what can such deep networks compute? How can we train them? How does information propagate through them? Why can they generalize? And how can we teach them to imagine? We review recent work in which methods of physical analysis rooted in statistical mechanics have begun to provide conceptual insights into these questions. These insights yield connections betw

Vol. 11:501-528 (Volume publication date March 2020) First published as a Review in Advance on December 9, 2019 https://doi.org/10.1146/annurev-conmatphys-031119-050745 Yasaman Bahri,1 Jonathan Kadmon,2 Jeffrey Pennington,1 Sam S. Schoenholz,1 Jascha Sohl-Dickstein,1 and Surya Ganguli1,2 1Google Brain, Google Inc., Mountain View, California 94043, USA 2Department of Applied Physics, Stanford University, Stanford, California 94035, USA; email: sganguli@stanford.edu The recent striking success of deep neural networks in machine learning raises profound questions about the theoretical principles