Explorative Modeling -- Unlocking a Third Pretraining Axis and End-to-End Generation | Alexi Gladstone
alexiglad.github.io · 3,613 words · saved by 2 readers
What in the world is Explorative Modeling?
Website: https://explorative-modeling.github.io/ GitHub: https://github.com/alexiglad/XM TLDR: We introduce Explorative Modeling, a new paradigm for generative modeling that acts as a third pretraining axis when added to existing generative models, and also enables end-to-end generation. Increasing exploration monotonically improves existing models across images, video, and language, and the gains grow with scale (7%→36% with data, 13%→23% with parameters). Concretely, Explorative Models (XMs) reach 6.2× sample efficiency, 4.1× FLOP efficiency, and 47% better parameter efficiency.…
saved by
related reading
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- [2404.02905] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictionarxiv.org
- True Agents Model the Worldprimeintellect.ai
- [1809.09087] Implicit Maximum Likelihood Estimationarxiv.org
- Video Generation Models Explosion 2024yenchenlin.me
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- 2409.02908arxiv.org
- Large Language Diffusion Modelsarxiv.org
- Esoteric Language Modelsarxiv.org
- Yang Songyang-song.net
- What are Diffusion Models?lilianweng.github.io
- GenAI Handbookgenai-handbook.github.io