Notes on the Origin of Implicit Regularization in SGD

I wanted to highlight an intriguing paper I presented at a journal club recently: Samuel L Smith, Benoit Dherin, David Barrett, Soham De (2021) On the Origin of Implicit Regularization in Stochastic Gradient DescentThere's actually a related paper that came out simultaneously, studying full-batch gradient descent instead of SGD: David

April 1, 2021 · deep learning generalization SGD differerntial equations Notes on the Origin of Implicit Regularization in SGD I wanted to highlight an intriguing paper I presented at a journal club recently: Samuel L Smith, Benoit Dherin, David Barrett, Soham De (2021) On the Origin of Implicit Regularization in Stochastic Gradient Descent There's actually a related paper that came out simultaneously, studying full-batch gradient descent instead of SGD: David G.T. Barrett, Benoit Dherin (2021) Implicit Gradient Regularization One of the most important insights in machine learning over

related reading

• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• Understanding deep learning requires rethinking generalizationarxiv.org
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• Just Ask for Generalization | Eric Jangevjang.com
• NeurIPS-2021-understanding-end-to-end-model-based-reinforcement-learning-methods-as-implicit-parameterization-Supplemental.pdflis.csail.mit.edu
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
• arxiv.org/pdf/1805.08522arxiv.org
• Why Does SGD Love Flat Minima? Marginally Better blogrishit-dagli.github.io
• The Generalization Mystery: Sharp vs Flat Minimainference.vc