✳flâneur — a map of the web's best reading

How can we solve diffuse threats like research sabotage with AI control?

redwoodresearch.substack.com · 2,144 words · saved by 1 readers

Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the original control paper or Ctrl-Z, because research sabotage differs from those settings in two crucial ways: Firstly, the AI can’t cause a catastrophe by behaving badly on just one occasion. To derail the entire project of solving alignment, the AI probably has to take a large number of malign actions. This contrasts with some other threat models (like rogue internal deployments), where the AI might succeed by acting badly on just a few occasions. Secondly and relatedly, individual bad actions in research sabotage aren’t very incriminating. If a model puts a bug in experiment code, or doesn’t think of an obvious idea, we won’t know whether it was malicious or just an honest mistake. This contrasts with other threat models -- if

How can we solve diffuse threats like research sabotage with AI control? Preventing research sabotage will require techniques very different from the original control paper. Vivek Hebbar Apr 30, 2025 10 Share Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the original control paper or Ctrl-Z , because research sabotage differs from those settings in two crucial ways: Firstly, the AI ca

Explore this link on the map →

related reading