A night-watchman ASI as a first step toward a great future – Unexpected Values

I took a week off from my day job of aligning AI to visit Forethought and think about the question: if we can align AI, what should we do with it? This post summarizes the state of my thinking at the end of that week. (The proposal described here is my own, and is not in any way endorsed by Forethought.) To my regular readers: this post has a more formal tone than my typical blog post (because I wrote it in a professional context) and is mostly targeted at my professional network, so it might require background that you don’t have. Sorry about that! Thanks to Mia Taylor, Tom Davidson, Ashwin Acharya, and a whole bunch of other people (mostly at Forethought) for discussion and comments. And a quick note: after writing this, I was told that Eric Drexler and David Dalrymple were thinking about a very similar idea in 2022, with essentially the same name. My thoughts here are independent of theirs. I expect the time right around when the first ASI gets built to be chaotic, unstable, and sca

I took a week off from my day job of aligning AI to visit Forethought and think about the question: if we can align AI, what should we do with it? This post summarizes the state of my thinking at the end of that week. (The proposal described here is my own, and is not in any way endorsed by Forethought.) To my regular readers: this post has a more formal tone than my typical blog post (because I wrote it in a professional context) and is mostly targeted at my professional network, so it might require background that you don’t have. Sorry about that! Thanks to Mia Taylor, Tom Davidson, Ashwin A