[2509.04259] RL's Razor: Why Online Reinforcement Learning Forgets Less

Abstract:Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL's Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.

View PDF HTML (experimental) Abstract:Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to…

saved by

• Jo J.
• Gilbert Chang

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Precise Manipulation with Efficient Online RLpi.website
• Self-Distillation Enables Continual Learningarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• RL is even more information inefficient than you thoughtdwarkesh.com
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org