[2212.09251] Discovering Language Model Behaviors with Model-Written Evaluations

Abstract:As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

[2212.09251] Discovering Language Model Behaviors with Model-Written Evaluations Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computation and Language arXiv:2212.09251 (cs) [Submitted on 19 Dec 2022] Title: Discovering Language Model Behaviors with Model-Written Evaluations Authors: Ethan Perez , Sam Ringer , Kamilė Lukošiūtė , Karina Nguyen , Edwin Chen , Scott Heiner , Craig Pettit , Catherine Olsson , Sandipan Kundu , Saurav Kadavath , Andy Jones , Anna Chen , Ben Mann , Bria

related reading

• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
• The bitter lesson of LLM evalsparsed.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• Eliciting Language Model Behaviors with Investigator Agents | Transluce AItransluce.org
• Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2303.13360] Towards the Scalable Evaluation of Cooperativeness in Language Modelsarxiv.org
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• Successful language model evals - Jason Weijasonwei.net
• [2502.13329] Language Models Can Predict Their Own Behaviorarxiv.org
• Language Modelinglena-voita.github.io