SFT, RL, and On-Policy Distillation Through a Distributional Lens | wh

I have been thinking about post-training methods in terms of distributions. A language model is a distribution over sequences. When we post-train it and attempt to teach it a task, we are reshaping this distribution. Different post-training methods differ in how they reshape this distribution, what they treat as the target and how directly they define this target.

I have been thinking about post-training methods in terms of distributions. A language model is a distribution over sequences. When we post-train it and attempt to teach it a task, we are reshaping this distribution. Different post-training methods differ in how they reshape this distribution, what they treat as the target and how directly they define this target. This is neither a very precise statement nor is it meant to be fully rigorous. I just find it to be a useful mental model, but I think it explains a lot of the qualitative differences between SFT, RL, and On-Policy Distillation. This

saved by

• Benedict Neo
• Henry Zhang
• Ishaan Panigrahi
• Tom Zheng
• Lailah Nabegu

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• will brown on X: "On SFT, RL, and on-policy distillation" / Xx.com
• Dense, on-policy, or both?baseten.co
• State of RL for reasoning LLMs | A. Weersaweers.de
• aiesi_post-training_public.pdfkawine.github.io
• Distillation Walkthroughvladfeinberg.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Learning to Remember — Lawrence Fengar-forum.github.io
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• On-Policy Distillation: Promise, Pitfalls, and Prospectslouieworth.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org