[2112.00861] A General Language Assistant as a Laboratory for Alignment

Abstract:Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

A General Language Assistant as a Laboratory for Alignment Amanda Askell∗ Yuntao Bai∗ Anna Chen∗ Dawn Drain∗ Deep Ganguli∗ Tom Henighan† Andy Jones† Nicholas Joseph† Ben Mann∗ Nova DasSarma Nelson Elhage arXiv:2112.00861v3 [cs.CL] 9 Dec 2021 Zac Hatfield-Dodds Danny Hernandez Jackson Kernion…

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• [2302.08582] Pretraining Language Models with Human Preferencesarxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Auditing language models for hidden objectives — LessWronglesswrong.com
• Alignment faking in large language modelsarxiv.org
• 2401.10020.pdfarxiv.org
• 2308.03958arxiv.org