[2204.14146] Training Language Models with Language Feedback

Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: comparisons between pairs of model-generated task outputs. Comparison feedback conveys limited information about human preferences per human evaluation. Here, we propose to learn from natural language feedback, which conveys more information per human evaluation. We learn from language feedback on model outputs using a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements, finding that only large language models (175B parameters) do so. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization ability.

Training Language Models with Language Feedback Jérémy Scheurer1 Jon Ander Campos1 2 Jun Shern Chan1 Angelica Chen1 Kyunghyun Cho1 3 4 Ethan Perez1 1 New York University, University of the Basque Country, 3 Genentech, 4 CIFAR LMB 2…

related reading

• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2302.08582] Pretraining Language Models with Human Preferencesarxiv.org
• 2401.10020.pdfarxiv.org
• 2308.03958arxiv.org
• 2005.14165arxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Language Models can Solve Computer Tasksarxiv.org