Explorative Modeling -- Unlocking a Third Pretraining Axis and End-to-End Generation | Alexi Gladstone
alexiglad.github.io · 3,613 words · saved by 2 readers
What in the world is Explorative Modeling?
Website: https://explorative-modeling.github.io/ GitHub: https://github.com/alexiglad/XM TLDR: We introduce Explorative Modeling, a new paradigm for generative modeling that acts as a third pretraining axis when added to existing generative models, and also enables end-to-end generation. Increasing exploration monotonically improves existing models across images, video, and language, and the gains grow with scale (7%→36% with data, 13%→23% with parameters). Concretely, Explorative Models (XMs) reach 6.2× sample efficiency, 4.1× FLOP efficiency, and 47% better parameter efficiency.…
saved by
related reading
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- True Agents Model the Worldprimeintellect.ai
- [2404.02905] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictionarxiv.org
- Video Generation Models Explosion 2024yenchenlin.me
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Zipfian grokking | Jasper Gilleyjagilley.github.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Just Ask for Generalization | Eric Jangevjang.com
- Explore | alphaXivalphaxiv.org
- Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
- Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
- Generative modelling in latent space – Sander Dielemansander.ai