✳flâneur — a map of the web's best reading

Diffusion is not necessarily Spectral Autoregression | Fabian Falck

fabianfalck.com · 7,234 words · saved by 1 readers

As the title suggests, this blog post directly responds to Sander Dieleman’s blog post ‘Diffusion is spectral autoregression’ which I really enjoyed. I spoke to many researchers in the community about it, and it prompted me to think about the relationship between Large Language Models (LLMs) and diffusion models, and why each excels on different tasks and data modalities. This blog post is accompanied by a paper titled ‘A Fourier Space Perspective on Diffusion Models’ together with my co-authors at Microsoft Research Cambridge that I cite at the very end of the post, and from which the majority of content is drawn from. Even though the focus of this blog post is slightly different, I will directly paraphrase from it where convenient. I begin by summarising Sander’s post from my perspective, qualitatively using exactly his argument, but trying to augment it with my own discussion and some new figures. Up front, I want to explain the notion of approximate spectral autoregression, because

Diffusion is not necessarily Spectral Autoregression tl;dr. DDPM diffusion models perform approximate autoregression in the Fourier domain, generating components from low to high frequency, but this inductive bias is not a necessity: diffusion models without any frequency hierarchy can perform equally well, and demonstrate improved high-frequency generation quality. In this blog post, we want to answer the question: Research question. Is approximate spectral autoregression , noising high-frequency components first, and consequently generating low-frequency before high-frequency components, a n

Explore this link on the map →

saved by

related reading