How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWrong

Authors: Dylan Xu, Alek Westover, Vivek Hebbar, Sebastian Prasanna, Nathan Sheffield, Buck Shlegeris, Julian Stastny …

x How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWrong AI Frontpage 62 How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? by Dylan Xu , Alek Westover , Vivek Hebbar , SebastianP , frisby , Julian Stastny 20th Apr 2026 24 min read 5 62 Authors: Dylan Xu, Alek Westover, Vivek Hebbar, Sebastian Prasanna, Nathan Sheffield, Buck Shlegeris, Julian Stastny Thanks to Eric Gan and Aghyad Deeb for feedback on a draft of this post. EDIT: After further consideration and @nostal

saved by

• Elliott Thornley

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• Model organisms researchers should check whether high LRs defeat their model organisms — LessWronglesswrong.com
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• How far does alignment midtraining generalize?alignment.openai.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Dataalignment.anthropic.com
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Simulated Users & Sad LLMs1a3orn.com