An Alternative to Test-Time Scaling
Exploring conditional computation and dynamic depth in language models. Contents Conditional Computation Width vs Depth O1 / Test-Time Scaling SMoE Dropout MoEUT Next Steps Conditional Computation I am interested in the idea of retrofitting existing language models to different architectural fo...
An Alternative to Test-Time Scaling Exploring conditional computation and dynamic depth in language models. Contents Conditional Computation Width vs Depth O1 / Test-Time Scaling SMoE Dropout MoEUT Next Steps Conditional Computation I am interested in the idea of retrofitting existing language models to different architectural formats to better take advantage of conditional computation. Most MoE literature tends to focus on creating a more efficient way to pretrain models at scale, but it's rarely used to create models that are capable of changing their computational width/depth dynamically. W
Explore this link on the map →related reading
- (Some of) The Models, They Just Don't Want to Learn | Tildeblog.tilderesearch.com
- How To Scale Your Modeljax-ml.github.io
- Composer2.pdfcursor.com
- Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- [2404.02258] Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelsarxiv.org
- [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
- [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
- On neural scaling and the quanta hypothesisericjmichaud.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- The Little Book of Deep Learningfleuret.org
- More Efficient In-Context Learning with GLaMblog.research.google