Alignment & Succession: The Two Bars of Alignment

Why much AI alignment discourse automatically treats the transition to powerful AI as a succession problem

Rembrandt, Saul and David When people talk about AI being “aligned”, I think there’s a lot of conflation between two different bars of success: The AI does what you say and does not go rogue. If you ask it to do a machine learning experiment for you, it actually does that instead of scheming against you and escaping onto the internet. If you say “stop”, it stops. (Some AI properties considered important for achieving this bar are corrigibility, non-deceptiveness, and intent alignment.) The AI has fully internalized our values, and could run society, and the lives of the humans in it, in a…

saved by

• Timothy Kostolansky
• Jason Hausenloy
• Ishaan Panigrahi

related reading

• Alignment & Succession: The Ideology of Successionnosetgauge.com
• The Case Against AI Control Research — LessWronglesswrong.com
• The Artificiality of Alignmentjoinreboot.org
• Alignment remains a hard, unsolved problem — LessWronglesswrong.com
• Alignment & Succession: Morality Lives in the Human Individualnosetgauge.com
• The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably — LessWronglesswrong.com
• What failure looks like — LessWronglesswrong.com
• How might we safely pass the buck to AI? — LessWronglesswrong.com
• What failure looks like — AI Alignment Forumalignmentforum.org
• The Artificial Intelligence Revolution: Part 2 - Wait But Whywaitbutwhy.com
• A Long Sequence of Small, Correct Decisionscalebwatney.substack.com
• The Problem — LessWronglesswrong.com