[2606.04075] Large Language Models Hack Rewards, and Society

Abstract:Reinforcement learning (RL) has become a dominant post-training paradigm, enabling large language models (LLMs) to learn from rewards. We observe that societal regulations are structurally similar to reward functions. They define measurable outcomes, thresholds, and exceptions, while often leaving institutional intent only partially specified. We hypothesise that the RL training process may exploit these gaps and therefore ask whether models' well-known tendency to hack reward functions during RL can scale into a more consequential failure mode named societal hacking: discovering loopholes in the rules society runs on. To study this phenomenon, we introduce SocioHack, a sandbox of 72 societal environments, and find that within these environments, reward hacking naturally emerges and leads to regulatory loophole discovery. Models learn to hack the social rules and generate strategies that remain technically compliant while defeating regulatory intent, and current LLM safeguards provide only limited mitigation. Therefore, collecting in-the-wild feedback for model training requires greater caution, and we need a next-generation post-training paradigm for safely iterating LLMs in real society.=

View PDF HTML (experimental) Abstract:Reinforcement learning (RL) has become a dominant post-training paradigm, enabling large language models (LLMs) to learn from rewards. We observe that societal regulations are structurally similar to reward functions. They define measurable outcomes, thresholds, and exceptions, while often leaving institutional intent only partially specified. We hypothesise that the RL training process may exploit these gaps and therefore ask whether models' well-known tendency to hack reward functions during RL can scale into a more consequential failure mode named…

saved by

• Benedict Neo

related reading

• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• [2606.12016] Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationarxiv.org
• Reward hacking is becoming more sophisticated and deliberate in frontier LLMs — LessWronglesswrong.com
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Simulated Users & Sad LLMs1a3orn.com
• Reward hacking behavior can generalize across tasks — AI Alignment Forumalignmentforum.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimesarxiv.org