Towards a scale-free theory of intelligent agency — AI Alignment Forum

I recently left OpenAI to pursue independent research. I’m working on a number of different research directions, but the most fundamental is my pursu…

x Towards a scale-free theory of intelligent agency — AI Alignment Forum Understanding systematization AI Frontpage 2025 Top Fifty: 14 % 39 Towards a scale-free theory of intelligent agency by Richard_Ngo 21st Mar 2025 Linkpost for www.mindthefuture.info 16 min read 51 39 I recently left OpenAI to pursue independent research. I’m working on a number of different research directions, but the most fundamental is my pursuit of a scale-free theory of intelligent agency. In this post I give a rough sketch of how I’m thinking about that. I’m erring on the side of sharing half-formed ideas, so there

saved by

• Lydia Nottingham

related reading

• The hot mess theory of AI misalignment: More intelligent agents behave less coherently | Jascha’s blogsohl-dickstein.github.io
• Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
• [1902.09469] Embedded Agencyarxiv.org
• ⿻ Symbiogenesis vs. Convergent Consequentialism — LessWronglesswrong.com
• Optimality is the tiger, and agents are its teeth — LessWronglesswrong.com
• Solipsistic Superintelligence is Unlikely to be Cooperativearxiv.org
• A positive case for how we might succeed at prosaic AI alignment — AI Alignment Forumalignmentforum.org
• Embedded Agency (full-text version) — LessWronglesswrong.com
• [2411.00114] Project Sid: Many-agent simulations toward AI civilizationarxiv.org
• Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWronglesswrong.com
• Everett branches, inter-light cone trade and other alien matters: Appendix to “An ECL explainer” — EA Forumforum.effectivealtruism.org
• UDT shows that decision theory is more puzzling than ever — LessWronglesswrong.com