[2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
Abstract:Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired. Yet existing data mixing methods address only one phase of this lifecycle at a time: some require smaller proxy models tied to a single training phase, others assume a fixed domain set, and continual learning lacks principled guidance altogether. We argue that data mixing is fundamentally an online decision making problem -- one that recurs throughout training and demands a single, unified solution. We introduce OP-Mix (On-Policy Mix), a data mixing algorithm that operates across the entire language model training lifecycle. Our main insight is that candidate data mixtures can be cheaply simulated by interpolating between low-rank adapters trained directly on the current model, eliminating separate proxy models and ensuring the search is always grounded in the model's actual learning dynamics. Across pretraining, continual midtraining, and continual instruction tuning, OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of the baselines. In pretraining, OP-Mix improves upon training without mixing by 6.3% in average perplexity. For continual learning, OP-Mix matches the performance of both retraining and on-policy distillation while using 66% and 95% less overall compute, respectively. OP-Mix suggests a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data.
# link_1r9i8on9i94.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Michael Y. Hu; Apurva Gandhi; Kyunghyun Cho; Tal Linzen; Pratyusha Sharma - Creator=arXiv GenPDF (tex2pdf:a6404ea) - Custom.DOI=https://doi.org/10.48550/arXiv.2605.15220 - Custom.License=http://creativecommons.org/licenses/by/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2605.15220v1 - Producer=pik
Explore this link on the map →saved by
related reading
- [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingresearch.nvidia.com
- DataRater: Meta-Learned Dataset Curationarxiv.org
- Why We Need Continual Learning | Andreessen Horowitza16z.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Self-Adapting Language Modelsarxiv.org
- Data Management For Large Language Models: A Surveyarxiv.org
- Composer2.pdfcursor.com
- The Continual Learning Problemjessylin.com