RL with KL penalties is better seen as Bayesian inference — LessWrong

This blog post is largely based on an EMNLP paper with Ethan Perez and Chris Buckley. It also benefited from discussions with and comments from Hady Elsahar, Germán Kruszewski, Marc Dymetman and Jérémy Scheurer. TLDR: KL-regularised RL, widely used as part of RL from human feedback (RLHF), is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function. The Bayesian perspective makes it clear that KL penalties aren’t a hack; they have a principled justification. It also nicely separates the modelling problem (defining a target distribution specifying the desired behaviour of an LM) and the inference problem (approximating that target distribution). Finally, it suggests that RL is not a good formal framework for thinking about LM alignment. Large language models (LMs) tend to generate outputs that reflect undesirable features of their training data such as offensiveness, social b

x RL with KL penalties is better seen as Bayesian inference — LessWrong Bayes' Theorem GPT Language Models (LLMs) Outer Alignment Reinforcement learning AI Frontpage 115 RL with KL penalties is better seen as Bayesian inference by Tomek Korbak , Ethan Perez 25th May 2022 AI Alignment Forum 15 min read 17 115 Ω 59 This blog post is largely based on an EMNLP paper with Ethan Perez and Chris Buckley. It also benefited from discussions with and comments from Hady Elsahar, Germán Kruszewski, Marc Dymetman and Jérémy Scheurer. TLDR : KL-regularised RL, widely used as part of RL from human feedback (

related reading

• RLHF Bookrlhfbook.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org
• Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
• rlhfbook.com/book.pdfrlhfbook.com
• Reinforcement Learning for Knowledge Awareness – kalomaze's kalomazing blogkalomaze.bearblog.dev