[2409.12822] Language Models Learn to Mislead Humans via RLHF

Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY" since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-art approach for detecting Intended Sophistry (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.

View PDF HTML (experimental) Abstract:Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY" since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and…

saved by

• Jacob G-W

related reading

• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Auditing language models for hidden objectives — LessWronglesswrong.com
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2302.08582] Pretraining Language Models with Human Preferencesarxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• Alignment faking in large language modelsarxiv.org