why assume AGIs will optimize for fixed goals? — LessWrong

When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted: …

x why assume AGIs will optimize for fixed goals? — LessWrong Goal-Directedness AI Frontpage 161 [ Question ] why assume AGIs will optimize for fixed goals? by nostalgebraist 10th Jun 2022 AI Alignment Forum 5 min read A 8 61 161 Ω 45 When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted: An AGI has a single terminal goal [1] . The goal is a fixed part of the AI's structure. The internal dynamics of the AI, if left to their own devices, will never modify the goal. The "outermost loop" of the AI's internal dynamics is

related reading

• Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
• Reward is not the optimization target — LessWronglesswrong.com
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
• wrapper-minds are the enemy — LessWronglesswrong.com
• How To Go From Interpretability To Alignment: Just Retarget The Search — LessWronglesswrong.com
• Optimality is the tiger, and agents are its teeth — LessWronglesswrong.com
• Why AIs aren't power-seeking yet — LessWronglesswrong.com
• Embedded Agency (full-text version) — LessWronglesswrong.com
• What Is The Alignment Problem? — LessWronglesswrong.com
• A central AI alignment problem: capabilities generalization, and the sharp left turn — LessWronglesswrong.com
• Alignment will happen by default. What’s next? — LessWronglesswrong.com