Stochastic gradient descent - Wikipedia

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the very high computational burden, achieving faster iterations in exchange for a lower convergence rate.[1]

Stochastic gradient descent - Wikipedia Jump to content From Wikipedia, the free encyclopedia Optimization algorithm Part of a series on Machine learning and data mining Paradigms Supervised learning Unsupervised learning Semi-supervised learning Self-supervised learning Reinforcement learning Meta-learning Online learning Batch learning Curriculum learning Rule-based learning Neuro-symbolic AI Neuromorphic engineering Quantum machine learning Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation Anomaly detection Data cleaning AutoML Ass

saved by

• Michael Domarkas

related reading

• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
• An overview of gradient descent optimization algorithmsruder.io
• Gradient descent - Wikipediaen.wikipedia.org
• A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
• [1512.04202] Preconditioned Stochastic Gradient Descentarxiv.org
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• Why Momentum Really Worksdistill.pub
• Highly optimized optimizers - by Ben Recht - arg minargmin.net
• Notes on the Origin of Implicit Regularization in SGDinference.vc
• Why Does SGD Love Flat Minima? Marginally Better blogrishit-dagli.github.io
• [2605.15522] Stochastic Non-Smooth Convex Optimization with Unbounded Gradientsarxiv.org
• [1511.06251] Stochastic modified equations and adaptive stochastic gradient algorithmsarxiv.org