[2602.12275] On-Policy Context Distillation for Language Models

Abstract:Context distillation enables language models to internalize in-context knowledge into their parameters. In our work, we propose On-Policy Context Distillation (OPCD), a framework that bridges on-policy distillation with context distillation by training a student model on its own generated trajectories while minimizing reverse Kullback-Leibler divergence against a context-conditioned teacher. We demonstrate the effectiveness of OPCD on two important applications: experiential knowledge distillation, where models extract and consolidate transferable knowledge from their historical solution traces, and system prompt distillation, where models internalize beneficial behaviors encoded in optimized prompts. Across mathematical reasoning, text-based games, and domain-specific tasks, OPCD consistently outperforms baseline methods, achieving higher task accuracy while better preserving out-of-distribution capabilities. We further show that OPCD enables effective cross-size distillation, where smaller student models can internalize experiential knowledge from larger teachers.

On-Policy Context Distillation for Language Models Tianzhu Ye∗ Li Dong∗ Xun Wu Shaohan Huang Furu Wei Microsoft Research https://aka.ms/GeneralAI arXiv:2602.12275v2 [cs.CL] 23 Mar 2026 Context distillation…

saved by

• Linda

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• [2604.00626] A Survey of On-Policy Distillation for Large Language Modelsarxiv.org
• [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2604.13010] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationarxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2606.04071] Covert Influence Between Language Modelsarxiv.org
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• [2605.10889] Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whyarxiv.org
• On the Geometry of On-Policy Distillationarxiv.org
• On-Policy Distillation: Promise, Pitfalls, and Prospectslouieworth.github.io