When does training a model change its goals?

Can a scheming AI's goals really stay unchanged through training?

Here are two opposing pictures of how training interacts with deceptive alignment: “goal-survival hypothesis”:1 When you subject a model to training, it can maintain its original goals regardless of what the training objective is, so long as it follows through on deceptive alignment (playing along with the training objective instrumentally). Even as it learns new skills and context-specific goals for doing well on the training objective, it continues to analyze these as instrumental to its original goals, and its values-upon-reflection aren’t affected by the learning process. “goal-change…

saved by

• Jo J.

related reading

• How training-gamers might function (and win)blog.redwoodresearch.org
• When does training a model change its goals?redwoodresearch.substack.com
• When does training a model change its goals? — LessWronglesswrong.com
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• New report: “Scheming AIs: Will AIs fake alignment during training in order to get power?”joecarlsmith.com
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Deep Deceptiveness — LessWronglesswrong.com
• Metagaming matters for training, evaluation, and oversightalignment.openai.com
• Alignment Faking Mitigationsalignment.anthropic.com
• Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com