mesaoptimizer - Schelling points in the AGI policy space

I’ve been thinking about memetically fit Schelling points in the AGI policy space. I’ll describe four such “Schelling policies”, and use them as pedagogical examples. MIRI’s new stated objective is the clearest example of a Schelling policy: “Shut it all down”. MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies. If you want to coordinate a large number of people coherently towards furthering a particular policy, “you get about five words” that you can make ‘common knowledge’ such that people can coordinate in a specific direction. The ease of communicating the policy makes a big difference in such conditions. When you attempt to communicate an idea widely, you’ll notice that people usually end up with multiple slightly (or sometimes wildly) differing

I’ve been thinking about memetically fit Schelling points in the AGI policy space. I’ll describe four such “Schelling policies”, and use them as pedagogical examples. MIRI’s new stated objective is the clearest example of a Schelling policy: “Shut it all down”. MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies. If you want to coordinate a large number of pe