Diffusion language models – Sander Dieleman
Diffusion models have completely taken over generative modelling of perceptual signals -- why is autoregression still the name of the game for language modelling? Can we do anything about that?
Diffusion models have completely taken over generative modelling of perceptual signals such as images, audio and video. Why is autoregression still the name of the game for language modelling? And can we do anything about that? Some thoughts about what it will take for other forms of iterative refinement to take over language modelling, the last bastion of autoregression. The rise of diffusion models Roughly three years ago, things were starting to look as if adversarial image generators were about to be supplanted by a powerful combination of autoregression and discrete representation learnin
Explore this link on the map →related reading
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- Diffusion is spectral autoregression – Sander Dielemansander.ai
- Large Language Diffusion Modelsarxiv.org
- Esoteric Language Modelsarxiv.org
- A Visual Guide to DiffusionGemma - by Maarten Grootendorstnewsletter.maartengrootendorst.com
- 2409.02908arxiv.org
- Diffusion is not necessarily Spectral Autoregression | Fabian Falckfabianfalck.com
- ⭐️ Diffusion Modelsandrewkchan.dev
- Generative modelling in latent space – Sander Dielemansander.ai
- Nemotron_Diffusion_Tech_Report_v1.pdfd1qx31qr3h6wln.cloudfront.net
- Large Language Diffusion Modelsarxiv.org
- Diffusion models from scratchchenyang.co