✳flâneur — a map of the web's best reading

Large Language Diffusion Models

ml-gsai.github.io · 849 words · saved by 1 readers

TL;DR: We introduce LLaDA, a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance. We contend that the intelligence of LLMs—manifested in scalability, instruction-following, in-context learning, conversational ability, and compression—stems not from the autoregressive mechanism per se, but rather from the core principle of generative modeling: approximating the true language distribution through maximum likelihood estimation. We introduce LLaDA (Large Language Diffusion with mAsking), a simple yet principled generative paradigm for large language models that demonstrates the aforementioned remarkable capabilities. LLaDA is a masked diffusion model [1, 2, 3] that follows standard pretraining and SFT while sampling via diffusion. During pretraining, it masks all tokens randomly at ratio 𝑡 ∼ 𝑈 [ 0 , 1 ] ; in SFT, only response tokens may be masked. The model simulates diffusion from full masking ( 𝑡 = 1 ) to unmasking ( 𝑡 =

Large Language Diffusion Models Large Language Diffusion Models A new paradigm for large language modeling based on diffusion Shen Nie 1 Fengqi Zhu 1 Zebin You 1 Xiaolu Zhang 2 Jingyang Ou 1 Jun Hu 2 Jun Zhou 2 Yankai Lin 1 Ji-Rong Wen 1 Chongxuan Li 1 1 Renmin University of China 2 Ant Group Paper Code 🤗 Model TL;DR: We introduce LLaDA, a diffusion model with an unprecedented 8B scale, trained entirely from scratch, rivaling LLaMA3 8B in performance. What is now proved was once only imagined. — William Blake Motivation We contend that the intelligence of LLMs—manifested in scalability , inst

Explore this link on the map →

saved by

related reading