Preventing Side-effects in Gridworlds

Can we ensure that artificial agents behave safely? Well, start at the bottom: We have not even solved the problem in the concrete 2D, fully-observable, finite case. Call this the “gridworld” case, following Sutton and Barto (1998). Recently, Google DeepMind released a game engine for building gridworlds, as well as a few examples of safety gridworlds - but these came without agents or featurisers. In April our team implemented RL agents for the engine, and started building a safety test suite for gridworlds. Our current progress can be found here, pending merge into the main repo. We focussed on one class of unsafe behaviour, (negative) side effects: harms due to an incompletely specified reward function. All real-world tasks involve many tacit secondary goals, from “…without breaking anything” to “…without being insulting”. But what prevents side effects? (Short of simply hand-coding the reward function to preclude them - which we can’t rely on, since that ad hoc approach won’t gener

Joint work with Karol Kubicki , Jessica Cooper and Tom McGrath at AISC 2018 . Can we ensure that artificial agents behave safely? Well, start at the bottom: We have not even solved the problem in the concrete 2D, fully-observable , finite case. Call this the “gridworld” case, following Sutton and Barto (1998) . Recently, Google DeepMind released a game engine for building gridworlds, as well as a few examples of safety gridworlds - but these came without agents or featurisers. In April our team implemented RL agents for the engine, and started building a safety test suite for gridworlds. Our c

related reading

• The Era of Experience Paper.pdfstorage.googleapis.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• pdfopenreview.net
• ROGUE:arxiv.org
• Reward Is Not Enough — LessWronglesswrong.com
• A Taxonomy of RL Environments for LLM Agentsleehanchung.github.io
• [AN #70]: Agents that help humans who are still learning about their own preferences — LessWronglesswrong.com
• Learning Beyond Gradientstrinkle23897.github.io
• How DeepMind's Generally Capable Agents Were Trained — LessWronglesswrong.com
• Deep Reinforcement Learning Doesn't Work Yetalexirpan.com
• Specification gaming: the flip side of AI ingenuity — Google DeepMinddeepmind.google
• The Iliad Intensive Course Materials — LessWronglesswrong.com