Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tür Hao Peng University of Illinois Urbana-Champaign {sagnikm3,lifan4,dilek,haopeng}@illinois.edu Abstract Reinforcement learning (RL) yields substantial improvements in large language models’ (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from updating only a small subnetwork comprising just 5%-30% of the parameters, with the rest effectively unchanged. We refer to this phenomenon as parameter update sparsity induced b
saved by
related reading
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
- Recent Advances in Language Model Fine-tuningruder.io