Introduction | RLHF and Post-Training Book by Nathan Lambert

A first-principles introduction to RLHF, what it changes in language models, and how it became part of modern post-training.

Reinforcement learning from human feedback (RLHF) is a technique used to incorporate human information into AI systems. RLHF emerged primarily as a method to solve hard-to-specify problems. With systems that are designed to be used by humans directly, such problems emerge all the time due to the often inexpressible nature of an individual’s preferences. This encompasses every domain of content and interaction with a digital system. RLHF’s early applications were often in control problems and other traditional domains for reinforcement learning (RL), where the goal is to optimize a specific…

saved by

• Laerdon Kim

related reading

• Checklists Are Better Than Reward Models For Aligning Language Modelsmachinelearning.apple.com
• rlhfbook.com/book.pdfrlhfbook.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF Bookrlhfbook.com
• RLHF | John Lambertjohnwlambert.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• How RLHF actually works - by Nathan Lambertinterconnects.ai
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• GitHub - ash80/RLHF_in_notebooks: RLHF (Supervised fine-tuning, reward model, and PPO) step-by-step in 3 Jupyter notebooksgithub.com
• Fine-tune Llama 2 with DPOhuggingface.co