[2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Abstract:Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

[2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback --> Computer Science > Computation and Language arXiv:2309.00267 (cs) [Submitted on 1 Sep 2023 ( v1 ), last revised 3 Sep 2024 (this version, v3)] Title: RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Authors: Harrison Lee , Samrat Phatale , Hassan Mansoor , Thomas Mesnard , Johan Ferret , Kellie Lu , Colton Bishop , Ethan Hall , Victor Carbune , Abhinav Rastogi , Sushant Prakash View a PDF of the paper titled RLAIF vs. RLHF: Scaling Reinforcement Learning from

saved by

• Lydia Nottingham
• N . Soma Sekhar

related reading

• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Constitutional AI: Harmlessness from AI Feedbackarxiv.org
• 2307.12950.pdfarxiv.org
• A simple technical explanation of RLH(AI)F | Kairos.fmkairos.fm
• 2401.10020.pdfarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc