Optimisation over non-stationary distributions creates weirder minds — LessWrong
TLDR: Sequentially mixing training objectives incentivises different training dynamics depending on the distinguishability of the training environmen…
x Optimisation over non-stationary distributions creates weirder minds — LessWrong AI Frontpage 36 Optimisation over non-stationary distributions creates weirder minds by Samuel Ratnam , Pjain 6th Jun 2026 4 min read 8 36 TLDR: Sequentially mixing training objectives incentivises different training dynamics depending on the distinguishability of the training environments and the amount of pressure for shared circuitry. We classify these patterns into three classes: ecological generalists, conditional policies, and strategy churn. We suggest that careful consideration of the pressures of non-st
saved by
related reading
- Do your capabilities homework — LessWronglesswrong.com
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWronglesswrong.com
- Composer2.pdfcursor.com
- Dense, on-policy, or both?baseten.co
- llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
- Narrow Misalignment is Hard, Emergent Misalignment is Easy — LessWronglesswrong.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- When does training a model change its goals?blog.redwoodresearch.org
- Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
- [2606.12016] Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationarxiv.org