Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models

Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models’ representations. Focusing on samples where RL succeeds but the base model fails,

Dan Shi Affiliation: TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China Email: shidan@tju.edu.cn Zhuowen Han Affiliation: TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China Email: dyxiong@tju.edu.cn Simon Ostermann Affiliation: German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany Affiliation: Saarland University, Saarbrücken, Germany Renren Jin Josef van Genabith Affiliation: German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany Affiliation: Saarland University,…

saved by

• Chris Shi

related reading

• Base Models Know How to Reason, Thinking Models Learn Whenarxiv.org
• Early Data Exposure Improves Robustness to Subsequent Fine-Tuningarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• DeepSeek-R1arxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
• [2606.07527] Post-training is (Massive) Supervised Learningarxiv.org
• RL's Razor: Why Online Reinforcement Learning Forgets Lessarxiv.org