Stochastic Gradient Descent SGD Lyapunov Convergence Proof Easy | oberman-lab

Explaining convergence of SGD in a self-contained article.

Introduction From a mathematical perspective, a neural network is just a parameterized function. To train a neural network is simply to minimize a function: Press enter or click to view image in full size We use the function f to abstract away the choice of the loss function. For example, we may want to minimize the mean-squared error of a fully-connected neural network with weights represented by w using input-output pairs (x,y): Press enter or click to view image in full size We use this notation because our analysis will not depend on the choice of the loss function, neural network…

related reading

• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• Gradient descent - Wikipediaen.wikipedia.org
• Why Momentum Really Worksdistill.pub
• Why Does SGD Love Flat Minima? Marginally Better blogrishit-dagli.github.io
• Effortless optimization through gradient flows – Machine Learning Research Blogfrancisbach.com
• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationsjmlr.org
• A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Notes on the Origin of Implicit Regularization in SGDinference.vc
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• [2604.06366] Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networksarxiv.org
• Linear regression: Gradient descent | Machine Learning | Google for Developersdevelopers.google.com