Regularization | RLHF Book by Nathan Lambert

Throughout the RLHF optimization, many regularization steps are used to prevent over-optimization of the reward model. Over-optimization in these contexts looks like models that output nonsensical text. Some examples of optimization “off the rails” are that models can output followable math reasoning with extremely incorrect answers, repeated text, switching languages, or excessive special characters. The most popular variant, used in most RLHF implementations at the time of writing, is a KL Distance from the current policy to a reference policy across the generated samples. Many other regularization techniques have emerged in the literature to then disappear in the next model iteration in that line of research. That is to say that regularization outside the core KL distance from generations is often used to stabilize experimental setups that can then be simplified in the next generations. Still, it is important to understand tools to constrain optimization in RLHF. The general formula

--> RLHF Book --> Regularization | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Regularization In this book we’ve learned many tools for modifying the model to learn from human preferences, verifiable rewards, and other valuable signals. All the methods we use are very powerful, and can cause the model to change too much relative to the strong, general model from the previous training stage (often called the reference model). When the model learns too much from

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• RL with KL penalties is better seen as Bayesian inference — LessWronglesswrong.com
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• rlhfbook.com/book.pdfrlhfbook.com
• RLHF Bookrlhfbook.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org