Approval-directed agency and the decision theory of Newcomb-like problems | Synthese | Springer Nature Link
Decision theorists disagree about how instrumentally rational agents, i.e., agents trying to achieve some goal, should behave in so-called Newcomb-like problems, with the main contenders being causal and evidential decision theory. Since the main goal of artificial intelligence research is to create machines that make instrumentally rational decisions, the disagreement pertains to this field. In addition to the more philosophical question of what the right decision theory is, the goal of AI poses the question of how to implement any given decision theory in an AI. For example, how would one go about building an AI whose behavior matches evidential decision theory’s recommendations? Conversely, we can ask which decision theories (if any) describe the behavior of any existing AI design. In this paper, we study what decision theory an approval-directed agent, i.e., an agent whose goal it is to maximize the score it receives from an overseer, implements. If we assume that the overseer rewards the agent based on the expected value of some von Neumann–Morgenstern utility function, then such an approval-directed agent is guided by two decision theories: the one used by the agent to decide which action to choose in order to maximize the reward and the one used by the overseer to compute the expected utility of a chosen action. We show which of these two decision theories describes the agent’s behavior in which situations.
Abstract Decision theorists disagree about how instrumentally rational agents, i.e., agents trying to achieve some goal, should behave in so-called Newcomb-like problems, with the main contenders being causal and evidential decision theory. Since the main goal of artificial intelligence research is to create machines that make instrumentally rational decisions, the disagreement pertains to this field. In addition to the more philosophical question of what the right decision theory is, the goal of AI poses the question of how to implement any given decision theory in an AI. For example, how…
saved by
related reading
- Decision Theory Research Overviewcasparoesterheld.com
- UDT shows that decision theory is more puzzling than ever — LessWronglesswrong.com
- TechnicalAgenda.pdfintelligence.org
- Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWronglesswrong.com
- Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
- Embedded Agency (full-text version) — LessWronglesswrong.com
- On Functional Decision Theory :: Wolfgang Schwarzumsu.de
- Towards a scale-free theory of intelligent agencymindthefuture.info
- Evidential decision theory - Wikipediaen.wikipedia.org
- Toward Idealized Decision Theoryarxiv.org
- Causal Decision Theory (Stanford Encyclopedia of Philosophy)plato.stanford.edu
- Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org