Preliminary Thoughts on Reward Hacking
Epistemic status: Early thoughts. Some ideas but no empirical testing or validation as yet. I’ve started thinking a fair bit about reward hacking recently. This is because frontier models are reportedly beginning to show signs of reward hacking especially for coding tasks. Thus, the era of easy-to-align pretraining-only models appears...
Epistemic status : Early thoughts. Some ideas but no empirical testing or validation as yet. I’ve started thinking a fair bit about reward hacking recently. This is because frontier models are reportedly beginning to show signs of reward hacking especially for coding tasks. Thus, the era of easy-to-align pretraining-only models appears to be coming to a close. Also in discussions and from my own experience, it seems to be that model’s reward hacking is one of the key bottlenecks that hinder recent reasoning RL models from continuing to improve is that they start to be able to hack the environm
Explore this link on the map →related reading
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- Reward hacking is becoming more sophisticated and deliberate in frontier LLMs — LessWronglesswrong.com
- Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
- Recent Frontier Models Are Reward Hacking - METRmetr.org
- (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL — LessWronglesswrong.com
- Reward hacking behavior can generalize across tasks — AI Alignment Forumalignmentforum.org
- Natural emergent misalignment from reward hacking \ Anthropicanthropic.com
- Systematic Reward Hacking and Prime Sprintsprimeintellect.ai
- Alignment remains a hard, unsolved problem — LessWronglesswrong.com
- Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI) — AI Alignment Forumalignmentforum.org
- Research Areas in Methods for Post-training and Elicitation (The Alignment Project by UK AISI) — AI Alignment Forumalignmentforum.org
- A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com