✳flâneur — a map of the web's best reading

How to prevent collusion when using untrusted models to monitor each other — AI Alignment Forum

alignmentforum.org · 8,574 words · saved by 1 readers

Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques, because by default you are unlikely to notice the problem until the hard-to-reverse outcome has already occurred. One of the main classes of techniques that you could apply in the hope of preventing these catastrophic actions is what we call untrusted monitoring (introduced in our paper here), which is roughly “before you execute a model’s action, ask another instance of that model whether the proposed action would be extremely dangerous”. This basic idea of asking one instance of a model to revi

x How to prevent collusion when using untrusted models to monitor each other — AI Alignment Forum AI Control Redwood Research AI Frontpage 42 How to prevent collusion when using untrusted models to monitor each other by Buck 25th Sep 2024 27 min read 13 42 Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming , and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltratin

Explore this link on the map →

related reading