[1511.06251] Stochastic modified equations and adaptive stochastic gradient algorithms

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

Stochastic modified equations and adaptive stochastic gradient algorithms Qianxiao Li∗1 , Cheng Tai†3,4 , and Weinan E‡2,3,4 1 Institute of High Performance Computing, Singapore 2…

related reading

• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationsjmlr.org
• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• Scalable Gradients for Stochastic Differential Equationsarxiv.org
• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu
• Why Momentum Really Worksdistill.pub
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Keep the gradient flowingfa.bianp.net
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• [1512.04202] Preconditioned Stochastic Gradient Descentarxiv.org
• An overview of gradient descent optimization algorithmsruder.io
• An SDE Framework for Adversarial Training, with Convergence and Robustness Analysisarxiv.org
• Highly optimized optimizers - by Ben Recht - arg minargmin.net