Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals | by DeepMind Safety Research | Medium
deepmindsafetyresearch.medium.com · 2,082 words · saved by 1 readers
By Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. For more details, check out…
Goal Misgeneralisation: Why Correct Specifications Aren’t Enough For Correct Goals DeepMind Safety Research 9 min read · Oct 7, 2022 -- 1 Listen Share By Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. For more details, check out our paper . As we build increasingly advanced AI systems, we want to make sure they don’t pursue undesired goals. This is the primary concern of the AI alignment community. Undesired behaviour in an AI agent is often the result of specification gaming —when the AI exploits an incorrectly specified reward. Howev
related reading
- Specification gaming: the flip side of AI ingenuity — Google DeepMinddeepmind.google
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- Reward is not the optimization target — LessWronglesswrong.com
- The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
- Models Don't "Get Reward" — LessWronglesswrong.com
- Metagaming matters for training, evaluation, and oversightalignment.openai.com
- How far does alignment midtraining generalize?alignment.openai.com
- Specification gaming: the flip side of AI ingenuity — Google DeepMinddeepmind.google
- Deep Deceptiveness — LessWronglesswrong.com
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com