ML Systems Will Have Weird Failure Modes

Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities [https://bounded-regret.ghost.io/p/1527e9dd-c48d-4941-9b14-4f7293318d5c/], and that thought experiments provide one approach [https://bounded-regret.ghost.io/p/a2d733a7-108a-4587-97fb-db90f66ce030/] towards predicting these capabilities and their consequences. In this post I’ll describe a particular thought experiment in detail.

Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities , and that thought experiments provide one approach towards predicting these capabilities and their consequences. In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that som

related reading

• What failure looks like — LessWronglesswrong.com
• What failure looks like — AI Alignment Forumalignmentforum.org
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
• Thought Experiments Provide a Third Anchorbounded-regret.ghost.io
• Emergent Deception and Emergent Optimizationbounded-regret.ghost.io
• Deep Deceptiveness — LessWronglesswrong.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover — LessWronglesswrong.com
• Simulated Users & Sad LLMs1a3orn.com
• In (highly contingent!) defense of interpretability-in-the-loop ML training — AI Alignment Forumalignmentforum.org
• Another (outer) alignment failure story — AI Alignment Forumalignmentforum.org