[2607.01763] Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Abstract:Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at this https URL.
View PDF HTML (experimental) Abstract:Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In…
saved by
related reading
- Self-Distillation Enables Continual Learningarxiv.org
- Reinforcement Learning via Self-Distillationarxiv.org
- Early Data Exposure Improves Robustness to Subsequent Fine-Tuningarxiv.org
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
- Self-Distillation Enables Continual Learningarxiv.org
- Self-Distillation Enables Continual Learningarxiv.org
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- [2605.10889] Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whyarxiv.org
- Dense, on-policy, or both?baseten.co
- [2604.00626] A Survey of On-Policy Distillation for Large Language Modelsarxiv.org
- [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org