flâneur

Why Does SGD Love Flat Minima? Marginally Better blog

rishit-dagli.github.io · 3,211 words · saved by 1 readers

This post is an intuitive look through the chronicles of Stochastic Gradient Descent (SGD), one of the most popular algorithms. I try to explain the reasoning behind some of the interesting aspects of SGD, like the stochastic gradient noise, the SGN covariance matrix, and SGD’s preference for flat minima. Recently, I’ve found it fascinating to explore and draw parallels with learning dynamics of algorithms based on gradient descent. Some ideas have particularly taken a bit of time for me to truly understand and with this, I hope to make it easier for others. Let’s take a quick look at Stochastic Gradient Descent (SGD). We consider the data samples 𝑥 = { 𝑥 𝑗 } 𝑗 = 1 𝑚 and similarly 𝑦 = { 𝑦 𝑗 } 𝑗 = 1 𝑚 , the model parameters 𝜃 , and some loss function 𝐿 ( 𝜃 , 𝑥 , 𝑦 ) . And following popular literature, for brevity, we denote the training loss as 𝐿 ( 𝜃 ) . We have SGD as follows: where 𝜂 is the step size, and 𝐿 ^ ( 𝜃 𝑡 , 𝑥 , 𝑦 ) is the stochastic estimate

Table of Contents ▼ But first… Modelling the Stochastic Gradient Noise (SGN) SGD can reach multiple minima What affects the SGN covariance matrix? SGD chooses a flat minima Concluding Thoughts Citation References Some things I learned from and some things I tried from a course at UofT. This article is a look through the chronicles of Stochastic Gradient Descent (SGD), one of the most popular algorithms. I try to explain the reasoning behind some of the interesting aspects of SGD, like the stochastic gradient noise, the SGN covariance matrix, and SGD’s preference for flat minima. Recently, I’ve

related reading