[1912.01683] Optimal Policies Tend to Seek Power

Abstract:Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

[1912.01683] Optimal Policies Tend to Seek Power --> Computer Science > Artificial Intelligence arXiv:1912.01683 (cs) [Submitted on 3 Dec 2019 ( v1 ), last revised 28 Jan 2023 (this version, v10)] Title: Optimal Policies Tend to Seek Power Authors: Alexander Matt Turner , Logan Smith , Rohin Shah , Andrew Critch , Prasad Tadepalli View a PDF of the paper titled Optimal Policies Tend to Seek Power, by Alexander Matt Turner and 4 other authors View PDF Abstract: Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pur

saved by

• Jacob G-W
• Anonymous Bat

related reading

• Reward is not the optimization target — LessWronglesswrong.com
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [2011.01859] A Study of Policy Gradient on a Class of Exactly Solvable Modelsarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [1606.05312] Successor Features for Transfer in Reinforcement Learningarxiv.org
• [2412.14135] Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivearxiv.org
• c21f4ce780c5c9d774f79841b81fdc6d-Paper.pdfproceedings.neurips.cc
• Complex behavior from intrinsic motivation to occupy future action-state path spacenature.com
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipearxiv.org