✳flâneur — a map of the web's best reading

There are no coherence theorems — AI Alignment Forum

alignmentforum.org · 10,416 words · saved by 1 readers

[Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum] For about fifteen years, the AI safety community has been discussing coherence arguments. In papers and posts on the subject, it’s often written that there exist 'coherence theorems' which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered that there are no such theorems. I’m concerned about this. Coherence arguments seem to be a moderately important part of the basic case for existential risk from AI. To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to

x There are no coherence theorems — AI Alignment Forum CAIS Philosophy Fellowship Midpoint Deliverables Coherence Arguments AI Frontpage 39 There are no coherence theorems by Dan H , Elliott Thornley 20th Feb 2023 23 min read 136 39 [Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum] Introduction For about fifteen years, the AI safety community has been discussing coherence arguments . In papers and posts on the subject, it’s often written that there exist 'coherence theorems' which state that, unless an agent can be represented as

Explore this link on the map →

related reading