✳flâneur — a map of the web's best reading

Win/continue/lose scenarios and execute/replace/audit protocols — AI Alignment Forum

alignmentforum.org · 2,212 words · saved by 1 readers

In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective. In brief: Consider a deployment of an AI in a setting where it’s going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we win, we lose, or the deployment continues (and the AI probably eventually attempts to attack again). So there’s two importantly different notions of attacks “failing”: we can catch the attack (in which case we win), or the attack can fail silently, either because it’s defective or because it’s defused by safety techniques, such that the attack doesn’t end up causing an unacceptable outcome but the AI also doesn’t get caught (in which case the deployment continues). When reasoning about how safe a protocol is, it’s crucial to distinguish between these two ways an attack can fail. And when designing protocols, it’s often worth

x Win/continue/lose scenarios and execute/replace/audit protocols — AI Alignment Forum AI Control Redwood Research AI Frontpage 33 Win/continue/lose scenarios and execute/replace/audit protocols by Buck 15th Nov 2024 8 min read 3 33 In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective . In brief: Consider a deployment of an AI in a setting where it’s going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we

Explore this link on the map →

related reading