Models Don't "Get Reward" - LessWrong
In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think…
x Models Don't "Get Reward" — LessWrong Best of LessWrong 2022 Reinforcement learning Distillation & Pedagogy Goal-Directedness AI Curated 349 Models Don't "Get Reward" by Sam Ringer 30th Dec 2022 AI Alignment Forum 6 min read 64 349 Ω 84 In terms of content, this has a lot of overlap with Reward is not the optimization target . I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight. When thinking about deception and RLHF training, a simplified threat model is something like this: A model takes some actions. If a human ap
saved by
related reading
- Reward is not the optimization target — LessWronglesswrong.com
- Reward Is Not the Optimization Targetturntrout.com
- The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
- Reward is not the optimization target — AI Alignment Forumalignmentforum.org
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- Reward Is Not Enough — LessWronglesswrong.com
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com
- A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
- Deep Deceptiveness — LessWronglesswrong.com
- Thomas Larsen's Shortform — LessWronglesswrong.com