On-Policy Distillation - Thinking Machines Lab
thinkingmachines.ai · 6,007 words · saved by 30 readers
On-policy, dense supervision is a useful tool for distillation
LLMs are capable of expert performance in focused domains, a result of several capabilities stacked together: perception of input, knowledge retrieval, plan selection, and reliable execution. This requires a stack of training approaches, which we can divide into three broad stages: Pre-training teaches general capacities such as language use, broad reasoning, and world knowledge. Mid-training imparts domain knowledge, such as code, medical databases, or internal company documents. Post-training elicits targeted behavior, such as instruction following, reasoning through math problems, or chat.
saved by
- Elizabeth Qiu
- Kaylee George
- Rajan Agarwal
- Ratan Kaliani
- Emma Guo
- Asher P
- Uzay Girit
- Thu Than
- Aaron Pham
- Katherine Driscoll
- Dhruv Gautam
- Sudarsh K
related reading
- [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- On-Policy Distillation: Promise, Pitfalls, and Prospectslouieworth.github.io
- Dense, on-policy, or both?baseten.co
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2604.00626] A Survey of On-Policy Distillation for Large Language Modelsarxiv.org
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
- ML Mentorship: Some Q/A about RLevjang.com
- will brown on X: "On SFT, RL, and on-policy distillation" / Xx.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org