Towards a scale-free theory of intelligent agency

An inherently multi-agent perspective

I recently left OpenAI to pursue independent research. I’m working on a number of different research directions, but the most fundamental is my pursuit of a scale-free theory of intelligent agency. In this post I give a rough sketch of how I’m thinking about that. I’m erring on the side of sharing half-formed ideas, so there may well be parts that don’t make sense yet. Nevertheless, I think this broad research direction is very promising. This post has two sections. The first describes what I mean by a theory of intelligent agency, and some problems with existing (non-scale-free) attempts.…

saved by

• Samuel Ratnam
• Lydia Nottingham
• Timothy Kostolansky
• Emil Ryd

related reading

• Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org
• Coasean Bargaining at Scaleblog.cosmos-institute.org
• The hot mess theory of AI misalignment: More intelligent agents behave less coherently | Jascha’s blogsohl-dickstein.github.io
• the trouble with making predictions as an agentsubstack.com
• Embedded Agency (full-text version) — LessWronglesswrong.com
• [1902.09469] Embedded Agencyarxiv.org
• Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
• Optimality is the tiger, and agents are its teeth — LessWronglesswrong.com
• TechnicalAgenda.pdfintelligence.org
• Approval-directed agency and the decision theory of Newcomb-like problemslink.springer.com
• Of Swarms and Sand Godsblog.cosmos-institute.org
• Orthogonality Thesis — LessWronglesswrong.com