Naturally Occurring Feedback is Common, Extractable and Useful

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Naturally Occurring Feedback is Common, Extractable and Useful Shachar Don-Yehiya 1 Leshem Choshen 2,3 Omri Abend 1 1 The Hebrew University of Jerusalem, 2 MIT, 3 MIT-IBM Watson AI Lab {first.last}@mail.huji.ac.il Abstract Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2204.14146] Training Language Models with Language Feedbackarxiv.org
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimesarxiv.org
• [2305.14387] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackarxiv.org
• 2401.10020.pdfarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• 2308.03958arxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io