✳flâneur — a map of the web's best reading
Musings on typicality – Sander Dieleman
sander.ai · 5,108 words · saved by 2 readers
A summary of my current thoughts on typicality, and its relevance to likelihood-based generative models.
If you’re training or sampling from generative models, typicality is a concept worth understanding. It sheds light on why beam search doesn’t work for autoregressive models of images, audio and video; why you can’t just threshold the likelihood to perform anomaly detection with generative models; and why high-dimensional Gaussians are “soap bubbles”. This post is a summary of my current thoughts on the topic. First, some context: one of the reasons I’m writing this, is to structure my own thoughts about typicality and the unintuitive behaviour of high-dimensional probability distributions. Mos
Explore this link on the map →saved by
related reading
- Musings on typicality – Sander Dielemanbenanne.github.io
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- Generative modelling in latent space – Sander Dielemansander.ai
- Language Modelinglena-voita.github.io
- Yang Songyang-song.net
- GenAI Handbookgenai-handbook.github.io
- Generative Modeling by Estimating Gradients of the Data Distribution | Yang Songyang-song.github.io
- 2409.02908arxiv.org
- Non-gaussian likelihoodarxiv.org
- Generating music in the waveform domain – Sander Dielemansander.ai
- Generative Modeling by Estimating Gradients of the Data Distribution | Yang Songyang-song.github.io
- Mode collapse - Wikipediaen.wikipedia.org