Rejection Sampling | RLHF Book by Nathan Lambert

The Reinforcement Learning from Human Feedback Book

--> RLHF Book --> Rejection Sampling | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Lecture 2: IFT, Reward Modeling, Rejection Sampling (Chap. 4, 5, & 9) Rejection Sampling Rejection Sampling (RS) is one of the most widely used yet least documented methods in preference fine-tuning. Many prominent RLHF papers use it as a core component of their training pipeline, yet no canonical implementation or explanation of why it works so well exists. RS can be applied at

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• rlhfbook.com/book.pdfrlhfbook.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• RLHF Bookrlhfbook.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• Models Don't "Get Reward" — LessWronglesswrong.com
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• [2510.13651] What is the objective of reasoning with reinforcement learning?arxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io