[2601.19897] Self-Distillation Enables Continual Learning

Abstract:Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations.

View PDF HTML (experimental) Abstract:Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT), a simple method that enables on-policy learning directly from…

saved by

• Dhruv Sheth
• Lev Chizhov
• Anonymous Cow

related reading

• Self-Distillation Enables Continual Learningarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Trainingarxiv.org
• Self-CTRL: Self-Consistency Training with Reinforcement Learningarxiv.org
• Flow Matching Policy Gradientsflowreinforce.github.io
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• Reinforcement Learning via Self-Distillationarxiv.org
• On-Policy Distillation: Promise, Pitfalls, and Prospectslouieworth.github.io
• [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• On the Geometry of On-Policy Distillationarxiv.org