Stochastic Gradient Descent SGD Lyapunov Convergence Proof Easy | oberman-lab

Explaining convergence of SGD in a self-contained article.

Introduction From a mathematical perspective, a neural network is just a parameterized function. To train a neural network is simply to minimize a function: Press enter or click to view image in full size We use the function f to abstract away the choice of the loss function. For example, we may want to minimize the mean-squared error of a fully-connected neural network with weights represented by w using input-output pairs (x,y): Press enter or click to view image in full size We use this notation because our analysis will not depend on the choice of the loss function, neural network…

related reading

• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• Why Momentum Really Worksdistill.pub
• Understanding the Neural Tangent Kernel – EigenTaleseigentales.com
• Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
• Effortless optimization through gradient flows – Machine Learning Research Blogfrancisbach.com
• Why Does SGD Love Flat Minima? Marginally Better blogrishit-dagli.github.io
• An overview of gradient descent optimization algorithmsruder.io
• Gradient descent - Wikipediaen.wikipedia.org
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
• CS231n Deep Learning for Computer Visioncs231n.github.io