✳flâneur — a map of the web's best reading

Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forum

alignmentforum.org · 8,135 words · saved by 1 readers

TL;DR: This document lays out the case for research on “model organisms of misalignment” – in vitro demonstrations of the kinds of failures that might pose existential threats – as a new and important pillar of alignment research. If you’re interested in working on this agenda with us at Anthropic, we’re hiring! Please apply to the research scientist or research engineer position on the Anthropic website and mention that you’re interested in working on model organisms of misalignment. We don’t currently have ~any strong empirical evidence for the most concerning sources of existential risk, most notably stories around dishonest AI systems that actively trick or fool their training processes or human operators: A significant part of why we think we don't see empirical examples of these failure modes is that they require that the AI develop several scary tendencies and capabilities such as situational awareness and deceptive reasoning and deploy them together. As a result, it seems usefu

x Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forum Best of LessWrong 2023 AI Evaluations Deceptive Alignment Deception Language Models (LLMs) Research Agendas AI Curated 124 Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research by evhub , Nicholas Schiefer , Carson Denison , Ethan Perez 8th Aug 2023 22 min read 30 124 TL;DR : This document lays out the case for research on “model organisms of misalignment” – in vitro demonstrations of the kinds of failures that might pose existential threats – as a new and importan

Explore this link on the map →

related reading