✳flâneur — a map of the web's best reading
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
research.nvidia.com · 1,349 words · saved by 1 readers
CLIMB is a clustering-based iterative data mixture bootstrapping method for language model pre-training.
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training 🧗🏻 CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training Shizhe Diao , Yu Yang , Yonggan Fu , Xin Dong , Dan Su , Markus Kliegl , Zijia Chen , Peter Belcak , Yoshi Suhara , Hongxu Yin , Mostofa Patwary , Yingyan (Celine) Lin , Jan Kautz , Pavlo Molchanov NVIDIA, Georgia Institute of Technology * Indicates Equal Contribution --> Paper 🤗 ClimbLab 🤗 ClimbMix Supplementary --> Code --> " target="_blank" class="external-link button is-normal is-rounded is-dark"> ar
Explore this link on the map →related reading
- [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
- GitHub - jacobhilton/deep_learning_curriculum: Language model alignment-focused deep learning curriculum · GitHubgithub.com
- Modern Pretraining Strategies: A Hands-On Guidetheneuralmaze.substack.com
- [2509.14786] Pre-training under infinite computearxiv.org
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
- Recent Advances in Language Model Fine-tuningruder.io
- Large Language Diffusion Modelsarxiv.org