[2212.09748] Scalable Diffusion Models with Transformers

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

View PDF HTML (experimental) Abstract:We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops -- through increased transformer depth/width or increased number of input tokens -- consistently have lower FID. In addition to possessing good scalability properties,…

saved by

• Quân Phạm Ngọc

related reading

• Scalable Diffusion Models with Transformersopenaccess.thecvf.com
• Fast Training of Diffusion Models with Masked Transformers | PDFarxiv.org
• What are Diffusion Models? | Lil'Loglilianweng.github.io
• What are Diffusion Models?lilianweng.github.io
• Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformersarxiv.org
• ⭐️ Diffusion Modelsandrewkchan.dev
• Large Language Diffusion Modelsarxiv.org
• 2408.11039arxiv.org
• The Illustrated Stable Diffusion – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and morenn.labml.ai
• [2506.14202] DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretationarxiv.org
• The Annotated Transformernlp.seas.harvard.edu