Scalable Gradients for Stochastic Differential Equations

The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differential equation whose solution is the gradient, a memory-efficient algorithm for caching noise, and conditions under which numerical solutions converge. In addition, we combine our method with gradient-based stochastic variational inference for latent stochastic differential equations. We use our method to fit stochastic dynamics defined by neural networks, achieving competitive performance on a 50-dimensional motion capture dataset.

Scalable Gradients for Stochastic Differential Equations Xuechen Li∗ Ting-Kam Leonard Wong Ricky T. Q. Chen David Duvenaud Google Research University of Toronto University of Toronto University of Toronto Vector Institute Vector Institute arXiv:2001.01328v6 [cs.LG] 18 Oct 2020…

saved by

• Roland Riachi

related reading

• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationsjmlr.org
• Density Dependent Singular Stochastic Differential Equationsarxiv.org
• On the Functional Levy-Ito Stochastic Calculusarxiv.org
• The Concentration of Invariant Measures for Stochastic Dynamical Systems with Locally Lipschitz Continuous Coefficients in R^darxiv.org
• MSR-TR-2009-192.pdfmicrosoft.com
• Stochastic Heat Equations with General Multiplicative Gaussian Noises: Hölder Continuity and Intermittencyarxiv.org
• [2008.13092] Fundamental Solution to 1D Degenerate Diffusion Equation with Locally Bounded Coefficientsarxiv.org
• Neural Manifold Ordinary Differential Equationsarxiv.org
• Comparison principle for stochastic heat equation on Rdmath.emory.edu
• A Feynman-Kac Approach for the Spatial Derivative of the Solution to the Wick Stochastic Heat Equation Driven by Time Homogeneous White Noisearxiv.org
• Functional Equations for the Stochastic Exponentialarxiv.org
• Hölder-continuity for the nonlinear stochastic heat equation with rough initial conditionsmath.emory.edu