How to Build a Diffusion Language Model | Kuleshov Group
An introduction to diffusion language models and the research advances that underlie today's diffusion LLMs. We describe the building blocks of recent open-source models, starting from simple masking diffusion, and including techniques for iterative refinement, post-training, and variable-length generation. Material is adapted from workshop talks and lectures at <a href='https://realm-gen-workshop.github.io/'>ICLR 2026</a> and <a href='https://cfe.columbia.edu/content/mlss2'>MLSS 2026</a>.
Introduction: Autoregressive and Diffusion Language Models Two families of generative AI algorithms are widely used today. For continuous data such as images or video, the state-of-the-art approach is based on diffusion models. For discrete data such as text or code, the standard approach is instead autoregressive models. This article explores an alternative for discrete data, one built on the modern paradigm of diffusion. Mainstream language models are autoregressive: they generate tokens left-to-right, one at a time, each conditioned on the tokens before it. This approach is powerful,…
saved by
related reading
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- Large Language Diffusion Modelsarxiv.org
- 2406.07524arxiv.org
- What are Diffusion Models?lilianweng.github.io
- 2503.09573arxiv.org
- Esoteric Language Modelsarxiv.org
- Diffusion language models – Sander Dielemansander.ai
- A Visual Guide to DiffusionGemma - by Maarten Grootendorstnewsletter.maartengrootendorst.com
- 2409.02908arxiv.org
- ⭐️ Diffusion Modelsandrewkchan.dev
- Large Language Diffusion Modelsarxiv.org
- Diffusion model - Wikipediaen.wikipedia.org