Scalable Gradients for Stochastic Differential Equations

The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differential equation whose solution is the gradient, a memory-efficient algorithm for caching noise, and conditions under which numerical solutions converge. In addition, we combine our method with gradient-based stochastic variational inference for latent stochastic differential equations. We use our method to fit stochastic dynamics defined by neural networks, achieving competitive performance on a 50-dimensional motion capture dataset.

Scalable Gradients for Stochastic Differential Equations Xuechen Li∗ Ting-Kam Leonard Wong Ricky T. Q. Chen David Duvenaud Google Research University of Toronto University of Toronto University of Toronto Vector Institute Vector Institute arXiv:2001.01328v6 [cs.LG] 18 Oct 2020…

saved by

• Roland Riachi

related reading

• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationsjmlr.org
• Advanced stochastic analysishairer.org
• A Primer on Stochastic Partial Differential Equationsmath.utah.edu
• 1806.07366arxiv.org
• Stochastic Differential Equations, Deep Learning, and High-Dimensional PDEsmitmath.github.io
• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitarxiv.org
• [1511.06251] Stochastic modified equations and adaptive stochastic gradient algorithmsarxiv.org
• A Review of Modern Stochastic Modeling: SDE/SPDE Numerics, Data-Driven Identification, and Generative Methods with Applications in Biology and Epidemiologyarxiv.org
• Step-by-Step Diffusion: An Elementary Tutorialarxiv.org
• Differentiable Programming from Scratchthenumb.at
• AdaGrad - Cornell University Computational Optimization Open Textbook - Optimization Wikioptimization.cbe.cornell.edu