Deep Energy Estimator.pdf

Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the resulting deep energy estimator network (DEEN) is designed as products of experts. We present the utility of DEEN in learning the energy, the score function, and in single-step denoising experiments for synthetic and high-dimensional data. We also diagnose stability problems in the direct estimation of the score function that had been observed for denoising autoencoders.

Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the re

saved by

• 3475

related reading

• Denoising Density Estimatorsarxiv.org
• Extending Stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy imagesproceedings.neurips.cc
• [2007.13640] Solving Linear Inverse Problems Using the Prior Implicit in a Denoiserarxiv.org
• Regularized by Score Matching (LeCun)papers.nips.cc
• Unsupervised Learning with SURE.pdfarxiv.org
• Learning to be Bayesian without Supervisionpapers.nips.cc
• [2304.11751] Score-Based Diffusion Models as Principled Priors for Inverse Imagingarxiv.org
• Beyond Gaussian Noise: A Generalized Approach to Likelihood Analysis with Non-Gaussian Noiseiopscience.iop.org
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation | Yang Songyang-song.github.io
• Yang Songyang-song.net
• Bayesian Neural Networkscs.toronto.edu
• The Little Book of Deep Learningfleuret.org