✳flâneur — a map of the web's best reading

[2504.10374] Ctrl-Z: Controlling AI Agents via Resampling

arxiv.org · 679 words · saved by 1 readers

Abstract:Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system administration tasks, and evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code in this environment. This multi-step setting introduces new attack and defense dynamics, which we investigate in order to design novel control protocols that prevent safety failures without hindering the ability of non-malicious agents to perform useful work. We introduce a class of control protocols called resample protocols that dynamically take additional samples of certain actions. We find these protocols significantly improve on existing techniques by selectively blocking the AI agent from executing suspicious code and incriminating the agent by generating additional examples of dangerous behavior. We measure the tradeoff between attack prevention and usefulness; our best protocol combines resampling with analysis of previous steps, reducing the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent.

[2504.10374] Ctrl-Z: Controlling AI Agents via Resampling Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2504.10374 (cs) [Submitted on 14 Apr 2025] Title: Ctrl-Z: Controlling AI Agents via Resampling Authors: Aryan Bhatt , Cody Rushing , Adam Kaufman , Tyler Tracy , Vasil Georgiev , David Matolcsi , Akbir Khan , Buck Shlegeris View a PDF of the paper titled Ctrl-Z: Controlling AI Agents via Resampling, by Aryan Bhatt and 7 other authors View PDF Abstract: Co

Explore this link on the map →

related reading