Thoughts on Loss Landscapes and why Deep Learning works — LessWrong

There are essentially two fundamental questions in the science of deep learning: 1.) Why are models trainable? And 2.) Why do models generalize? The…

x Thoughts on Loss Landscapes and why Deep Learning works — LessWrong General intelligence Machine Learning (ML) AI Frontpage 54 Thoughts on Loss Landscapes and why Deep Learning works by beren 25th Jul 2023 21 min read 4 54 Crossposted from my personal blog . Epistemic status : Pretty uncertain. I don’t have an expert level understanding of current views in the science of deep learning about why optimization works but just read papers as an amateur. Some of the arguments I present here might be already either known or disproven. If so please let me know! There are essentially two fundamental

related reading

• Thoughts on loss landscapes and why deep learning worksberen.io
• Just Ask for Generalization | Eric Jangevjang.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Understanding deep learning requires rethinking generalizationarxiv.org
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• A Recipe for Training Neural Networkskarpathy.github.io
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• The Generalization Mystery: Sharp vs Flat Minimainference.vc
• Neural networks generalize because of this one weird trick — LessWronglesswrong.com
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• A Recipe for Training Neural Networkskarpathy.github.io