Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise Xuanqing Liu Tesi Xiao Department of Computer Science, UCLA Department of Statistics, UC Davis xqliu@cs.ucla.edu texiao@ucdavis.edu arXiv:1906.02355v1 [cs.LG] 5 Jun 2019…

related reading

• Neural Ordinary Differential Equationsarxiv.org
• 1806.07366arxiv.org
• Stochastic Differential Equations, Deep Learning, and High-Dimensional PDEsmitmath.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• An SDE Framework for Adversarial Training, with Convergence and Robustness Analysisarxiv.org
• Ilya 30u30arc.net
• A Recipe for Training Neural Networkskarpathy.github.io
• SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimatesarxiv.org
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitarxiv.org
• DiffEqFlux.jl – A Julia Library for Neural Differential Equationsjulialang.org
• A Review of Modern Stochastic Modeling: SDE/SPDE Numerics, Data-Driven Identification, and Generative Methods with Applications in Biology and Epidemiologyarxiv.org
• David Duvenaudcs.toronto.edu