Training Overview | RLHF Book by Nathan Lambert

The optimization of reinforcement learning from human feedback (RLHF) builds on top of the standard RL setup. In RL, an agent takes actions, 𝑎 , sampled from a policy, 𝜋 , with respect to the state of the environment, 𝑠 , to maximize reward, 𝑟 [1]. Traditionally, the environment evolves with respect to a transition or dynamics function 𝑝 ( 𝑠 𝑡 + 1 | 𝑠 𝑡 , 𝑎 𝑡 ) . Hence, across a finite episode, the goal of an RL agent is to solve the following optimization: 𝐽 ( 𝜋 ) = 𝐸 𝜏 ∼ 𝜋 [ ∑ 𝑡 = 0 ∞ 𝛾 𝑡 𝑟 ( 𝑠 𝑡 , 𝑎 𝑡 ) ] , ( 1 ) where 𝛾 is a discount factor from 0 to 1 that balances the desirability of near- versus future-rewards. Multiple methods for optimizing this expression are discussed in Chapter 11. A standard illustration of the RL loop is shown in fig. 1 and how it compares to fig. 2. There are multiple core changes from the standard RL setup to that of RLHF: Given the single-turn nature of the problem, the optimization can be re-written without the time

--> RLHF Book --> Training Overview | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Lecture 0: Prerequisites Lecture 1: Overview (Chapters 1–3) Training Overview In this chapter we provide a cursory overview of RLHF training, before getting into the specifics later in the book. RLHF, while optimizing a simple loss function, involves training multiple, different AI models in sequence and then linking them together in a complex, online optimization. Here, we introd

related reading

• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• rlhfbook.com/book.pdfrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• The N Implementation Details of RLHF with PPO | ICLR Blogposts 2024iclr-blogposts.github.io
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• How RLHF actually works - by Nathan Lambertinterconnects.ai