CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

CLIMB is a clustering-based iterative data mixture bootstrapping method for language model pre-training.

CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training 🧗🏻 CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training Shizhe Diao , Yu Yang , Yonggan Fu , Xin Dong , Dan Su , Markus Kliegl , Zijia Chen , Peter Belcak , Yoshi Suhara , Hongxu Yin , Mostofa Patwary , Yingyan (Celine) Lin , Jan Kautz , Pavlo Molchanov NVIDIA, Georgia Institute of Technology * Indicates Equal Contribution --> Paper 🤗 ClimbLab 🤗 ClimbMix Supplementary --> Code --> " target="_blank" class="external-link button is-normal is-rounded is-dark"> ar

related reading

• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2509.14786] Pre-training under infinite computearxiv.org
• Shaping capabilities with token-level data filteringarxiv.org
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
• radford2018improving.pdfcs.ubc.ca
• Generalized Language Models | Lil'Loglilianweng.github.io