Settings – ControlArena
control-arena.aisi.org.uk · 6,383 words · saved by 1 readers
Open-source framework for AI Control evaluations
Settings – ControlArena Source Overview A Setting in ControlArena defines a complete evaluation environment for testing control protocols. Settings simulate realistic risk scenarios where an AI agent performs legitimate work (the main task ) while potentially attempting malicious objectives (the side task ). For example, in the original AI Control paper , agents solved programming problems from the APPS dataset as their main task, while side tasks involved inserting subtle backdoors that pass tests but fail on specific secret inputs. Settings are implemented by extending control_arena.settings
related reading
- GitHub - UKGovernmentBEIS/control-arena: ControlArena is a collection of settings, model organisms and protocols - for running control experiments. · GitHubgithub.com
- [2604.15384] LinuxArena: A Control Setting for AI Agents in Live Production Software Environmentsarxiv.org
- Demystifying evals for AI agents \ Anthropicanthropic.com
- BashArena and Control Setting Designblog.redwoodresearch.org
- The case for ensuring that powerful AIs are controlled — LessWronglesswrong.com
- Incident Report: unsanctioned agent behaviour during cyber testing | AISI Workaisi.gov.uk
- An overview of areas of control workblog.redwoodresearch.org
- The Case Against AI Control Research — LessWronglesswrong.com
- AI Control: Improving Safety Despite Intentional Subversion — LessWronglesswrong.com
- ARENA - AI Safety Curriculumlearn.arena.education
- Effective harnesses for long-running agents \ Anthropicanthropic.com
- General Agent: A Self-Evolving, Synthetic Agent Environmentprimeintellect.ai