[2509.04259] RL's Razor: Why Online Reinforcement Learning Forgets Less

Abstract:Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL's Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.

View PDF HTML (experimental) Abstract:Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to…

saved by

• Jo J.
• Gilbert Chang

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Self-Distillation Enables Continual Learningarxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Reinforcement Learning for Knowledge Awareness – kalomaze's kalomazing blogkalomaze.bearblog.dev
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• RLHF Bookrlhfbook.com