Musings on typicality – Sander Dieleman

A summary of my current thoughts on typicality, and its relevance to likelihood-based generative models.

If you’re training or sampling from generative models, typicality is a concept worth understanding. It sheds light on why beam search doesn’t work for autoregressive models of images, audio and video; why you can’t just threshold the likelihood to perform anomaly detection with generative models; and why high-dimensional Gaussians are “soap bubbles”. This post is a summary of my current thoughts on the topic. First, some context: one of the reasons I’m writing this, is to structure my own thoughts about typicality and the unintuitive behaviour of high-dimensional probability distributions. Mos

saved by

• Ashwin Reddy
• Noah
• Clem von Stengel

related reading

• Musings on typicality – Sander Dielemanbenanne.github.io
• Generative modelling in latent space – Sander Dielemansander.ai
• Generating music in the waveform domain – Sander Dielemansander.ai
• Diffusion is spectral autoregression – Sander Dielemansander.ai
• Tips for Training Likelihood Modelsblog.evjang.com
• Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
• What are Diffusion Models? | Lil'Loglilianweng.github.io
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Diffusion is not necessarily Spectral Autoregression | Fabian Falckfabianfalck.com
• Learning the integral of a diffusion model – Sander Dielemansander.ai
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Diffusion models from scratchchenyang.co