Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models Sagnik Mukherjee Lifan Yuan Dilek Hakkani-Tür Hao Peng University of Illinois Urbana-Champaign {sagnikm3,lifan4,dilek,haopeng}@illinois.edu Abstract Reinforcement learning (RL) yields substantial improvements in large language models’ (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from updating only a small subnetwork comprising just 5%-30% of the parameters, with the rest effectively unchanged. We refer to this phenomenon as parameter update sparsity induced b
saved by
related reading
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- State of RL for reasoning LLMs | A. Weersaweers.de
- RL's Razor: Why Online Reinforcement Learning Forgets Lessarxiv.org
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- LLM Resourcesforrestbicker.com