flâneur

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

arxiv.org · 5,972 words · saved by 1 readers

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices — that is, the manner in which a capability is acquired — shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure — mixing post-training data into pretraining — consistently improves the frontier between reta

Natalia Cerebro & Amelie P. Amygdale ††thanks: Use footnote for providing further information about author (webpage, alternative address)—not for acknowledging funding agencies. Funding acknowledgements go at the end of the paper. Affiliation: Department of Computer Science Affiliation: Cranberry-Lemon University Affiliation: Pittsburgh, PA 15213, USA Email: {hippo,brain,jen}@cs.cranberry-lemon.edu Ji Q. Ren & Yevgeny LeNet Affiliation: Department of Computational Neuroscience Affiliation: University of the Witwatersrand Affiliation: Joburg, South Africa Email: {robot,net}@wits.ac.za…

saved by

related reading