[1511.06251] Stochastic modified equations and adaptive stochastic gradient algorithms

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

Stochastic modified equations and adaptive stochastic gradient algorithms Qianxiao Li∗1 , Cheng Tai†3,4 , and Weinan E‡2,3,4 1 Institute of High Performance Computing, Singapore 2…

related reading

• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationsjmlr.org
• Scalable Gradients for Stochastic Differential Equationsarxiv.org
• [2605.15522] Stochastic Non-Smooth Convex Optimization with Unbounded Gradientsarxiv.org
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• [1512.04202] Preconditioned Stochastic Gradient Descentarxiv.org
• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• An SDE Framework for Adversarial Training, with Convergence and Robustness Analysisarxiv.org
• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
• [2604.06366] Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networksarxiv.org
• [2505.02621] Mirror Mean-Field Langevin Dynamicsarxiv.org