Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlow

This work is authored by Haoxiang Wang*, Wei Xiong*, Tengyang Xie, Han Zhao, Tong Zhang (* indicates equal contribution) Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we pro

Table of Contents Abstract Preliminaries RLHF Pipeline The Need for Interpretable Reward Models Multi-Objective Reward Modeling Meets Mixture-of-Experts Stage-1: Multi-Objective Reward Modeling Absolute-Rating Multi-Objective Reward Model (ArmoRM) Implementation of ArmoRM Stage-2: Mixture-of-Experts Aggregation of Reward Objectives ArmoRM with Mixture-of-Experts (Armo-MoE) Implementation of ArmoRM-MoE Empirical Results: SoTA on Reward-Bench Usage Example (Code Demo) Citation This work is authored by Haoxiang Wang* , Wei Xiong* , Tengyang Xie , Han Zhao , Tong Zhang (* indicates equal contribut

saved by

• Winnie Xu

related reading

• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF Bookrlhfbook.com
• rlhfbook.com/book.pdfrlhfbook.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org