Checklists Are Better Than Reward Models For Aligning Language Models - Apple Machine Learning Research

Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this --…

AuthorsVijay Viswanathan†, Yanchao Sun, Shuang Ma‡**, Xiang Kong, Meng Cao, Graham Neubig†, Tongshuang Wu† Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this — typically using fixed criteria such as “helpfulness” and “harmfulness”. In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose “Reinforcement Learning from Checklist Feedback” (RLCF). From instructions, we…

saved by

• Laerdon Kim

related reading

• user_interactions.pdfself-distillation.github.io
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• 2401.10020.pdfarxiv.org
• Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
• 2307.12950.pdfarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainsarxiv.org
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org