flâneur

Li Jiang | On-Policy Distillation: Promise, Pitfalls, and Prospects

louieworth.github.io · 2,730 words · saved by 1 readers

The promise of on-policy distillation (OPD) (Gu et al., 2023; Agarwal et al., 2023; Lu and Thinking Machines Lab, 2025) comes from how it rearranges two fundamental ingredients, the policy that generates rollouts and the density of the learning signal attached to those rollouts. The first axis is on-policy versus off-policy, asking whether we train on trajectories sampled from the student itself or from an external teacher or dataset. The second axis is sparse versus dense supervision, asking whether the model receives only an outcome-level reward or a token-level signal along the trajectory. Supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) occupy opposite corners of this space. SFT provides dense learning signals, but usually on off-policy trajectories. RLVR uses on-policy rollouts, but often reduces an entire reasoning trace to a sparse verifiable reward. OPD tries to combine the useful half of both, on-policy student rollouts with dense teacher

The promise of on-policy distillation (OPD) (Gu et al., 2023; Agarwal et al., 2023; Lu and Thinking Machines Lab, 2025) comes from how it rearranges two fundamental ingredients, the policy that generates rollouts and the density of the learning signal attached to those rollouts. The first axis is on-policy versus off-policy, asking whether we train on trajectories sampled from the student itself or from an external teacher or dataset. The second axis is sparse versus dense supervision, asking whether the model receives only an outcome-level reward or a token-level signal along the…

saved by

related reading