SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates

Uncertainty quantification is a fundamental yet unsolved problem for deep learning. The Bayesian framework provides a principled way of uncertainty estimation but is often not scalable to modern deep neural nets (DNNs) that have a large number of parameters. Non-Bayesian methods are simple to implement but often conflate different sources of uncertainties and require huge computing resources. We propose a new method for quantifying uncertainties of DNNs from a dynamical system perspective. The core of our method is to view DNN transformations as state evolution of a stochastic dynamical system and introduce a Brownian motion term for capturing epistemic uncertainty. Based on this perspective, we propose a neural stochastic differential equation model (SDE-Net) which consists of (1) a drift net that controls the system to fit the predictive function; and (2) a diffusion net that captures epistemic uncertainty. We theoretically analyze the existence and uniqueness of the solution to SDE-Net. Our experiments demonstrate that the SDE-Net model can outperform existing uncertainty estimation methods across a series of tasks where uncertainty plays a fundamental role.

SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates Lingkai Kong 1 Jimeng Sun 2 Chao Zhang 1 Abstract their predictions and produce mis-calibrated output proba- bilities for classification (Guo et al., 2017). Moreover, they Uncertainty…

related reading

• Bayesian Neural Networkscs.toronto.edu
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisearxiv.org
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitarxiv.org
• [1706.04599] On Calibration of Modern Neural Networksarxiv.org
• Denoising Density Estimatorsarxiv.org
• Scalable Gradients for Stochastic Differential Equationsarxiv.org
• [2604.06366] Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networksarxiv.org
• Stochastic Differential Equations, Deep Learning, and High-Dimensional PDEsmitmath.github.io
• Deep Energy Estimator.pdfarxiv.org
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• Stochastic Parameter Decompositionarxiv.org
• Statistical Mechanics of Deep Learningganguli-gang.stanford.edu