[1507.01986] Toward Idealized Decision Theory
Abstract:This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.
View PDF HTML (experimental) Abstract:This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research. Comments: This is an extended…
saved by
related reading
- [2512.15584] A Decision-Theoretic Approach for Managing Misalignmentarxiv.org
- Alignment remains a hard, unsolved problem — LessWronglesswrong.com
- Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
- TechnicalAgenda.pdfintelligence.org
- Approval-directed agency and the decision theory of Newcomb-like problemslink.springer.com
- On Functional Decision Theory :: Wolfgang Schwarzumsu.de
- The hot mess theory of AI misalignment: More intelligent agents behave less coherently | Jascha’s blogsohl-dickstein.github.io
- Logical decision theories — LessWrongarbital.com
- UDT shows that decision theory is more puzzling than ever — LessWronglesswrong.com
- Towards a scale-free theory of intelligent agencymindthefuture.info
- VDT: a solution to decision theory — LessWronglesswrong.com
- Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org