Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices — that is, the manner in which a capability is acquired — shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure — mixing post-training data into pretraining — consistently improves the frontier between reta

Natalia Cerebro & Amelie P. Amygdale ††thanks: Use footnote for providing further information about author (webpage, alternative address)—not for acknowledging funding agencies. Funding acknowledgements go at the end of the paper. Affiliation: Department of Computer Science Affiliation: Cranberry-Lemon University Affiliation: Pittsburgh, PA 15213, USA Email: {hippo,brain,jen}@cs.cranberry-lemon.edu Ji Q. Ren & Yevgeny LeNet Affiliation: Department of Computational Neuroscience Affiliation: University of the Witwatersrand Affiliation: Joburg, South Africa Email: {robot,net}@wits.ac.za…

saved by

• Chris Shi

related reading

• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Trainingarxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Learning to Remember — Lawrence Fengar-forum.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Shaping capabilities with token-level data filteringarxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org