Optimisation over non-stationary distributions creates weirder minds — LessWrong
TLDR: Sequentially mixing training objectives incentivises different training dynamics depending on the distinguishability of the training environmen…
x Optimisation over non-stationary distributions creates weirder minds — LessWrong AI Frontpage 36 Optimisation over non-stationary distributions creates weirder minds by Samuel Ratnam , Pjain 6th Jun 2026 4 min read 8 36 TLDR: Sequentially mixing training objectives incentivises different training dynamics depending on the distinguishability of the training environments and the amount of pressure for shared circuitry. We classify these patterns into three classes: ecological generalists, conditional policies, and strategy churn. We suggest that careful consideration of the pressures of non-st
saved by
related reading
- Alignment pretraining could backfire — LessWronglesswrong.com
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWronglesswrong.com
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- Reward is not the optimization target — LessWronglesswrong.com
- Models Don't "Get Reward" — LessWronglesswrong.com
- Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
- Non_Interactive – Software & MLnonint.com
- Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
- New report: “Scheming AIs: Will AIs fake alignment during training in order to get power?”joecarlsmith.com
- Generalization Dynamics of LM Pre-training — Jiaxin Wenjiaxin-wen.github.io
- Simulated Users & Sad LLMs1a3orn.com