Diffusion language models – Sander Dieleman

Diffusion models have completely taken over generative modelling of perceptual signals -- why is autoregression still the name of the game for language modelling? Can we do anything about that?

Diffusion models have completely taken over generative modelling of perceptual signals such as images, audio and video. Why is autoregression still the name of the game for language modelling? And can we do anything about that? Some thoughts about what it will take for other forms of iterative refinement to take over language modelling, the last bastion of autoregression. The rise of diffusion models Roughly three years ago, things were starting to look as if adversarial image generators were about to be supplanted by a powerful combination of autoregression and discrete representation learnin

related reading

• Diffusion is spectral autoregression – Sander Dielemansander.ai
• What are Diffusion Models? | Lil'Loglilianweng.github.io
• What are Diffusion Models?lilianweng.github.io
• Diffusion models are autoencoders – Sander Dielemansander.ai
• Generative modelling in latent space – Sander Dielemansander.ai
• Kuleshov Group | How to Build a Diffusion Language Modelkuleshov-group.github.io
• Diffusion will be everywherevarunneal.github.io
• Diffusion models from scratchchenyang.co
• Diffusion is not necessarily Spectral Autoregression | Fabian Falckfabianfalck.com
• Large Language Diffusion Modelsarxiv.org
• ⭐️ Diffusion Modelsandrewkchan.dev
• Learning the integral of a diffusion model – Sander Dielemansander.ai