Musings on typicality – Sander Dieleman

A summary of my current thoughts on typicality, and its relevance to likelihood-based generative models.

If you’re training or sampling from generative models, typicality is a concept worth understanding. It sheds light on why beam search doesn’t work for autoregressive models of images, audio and video; why you can’t just threshold the likelihood to perform anomaly detection with generative models; and why high-dimensional Gaussians are “soap bubbles”. This post is a summary of my current thoughts on the topic. First, some context: one of the reasons I’m writing this, is to structure my own thoughts about typicality and the unintuitive behaviour of high-dimensional probability distributions. Mos

saved by

• Ashwin Reddy
• Noah
• Clem von Stengel

related reading

• Musings on typicality – Sander Dielemanbenanne.github.io
• Generative modelling in latent space – Sander Dielemansander.ai
• Generating music in the waveform domain – Sander Dielemansander.ai
• Tips for Training Likelihood Modelsblog.evjang.com
• GenAI Handbookgenai-handbook.github.io
• The “it” in AI models is the dataset. — Non_Intnonint.com
• What are Diffusion Models?lilianweng.github.io
• Language Modelinglena-voita.github.io
• [1809.09087] Implicit Maximum Likelihood Estimationarxiv.org
• Yang Songyang-song.net
• 2409.02908arxiv.org
• Generative Modeling by Estimating Gradients of the Data Distribution | Yang Songyang-song.github.io