Generalization Dynamics of LM Pre-training — Jiaxin Wen
People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like modes, i.e. distinct algorithms implemented by distinct circuits. We call this mode-hopping. Across our suite, LMs can suddenly latch onto memorized or in-context patterns instead of in-context learning, use System 1 instead of System 2 thinking, pick up what sounds true instead of what is true, fail at multi-hop persona QA, out-of-context reasoning, and emergent misalignment — then just as suddenly revert and generalize. Mode-hopping is not explained by standard optimization dynamics: it is locally stable and can not be fixed by checkpoint averaging. We instead think of it as a capacity allocation problem: in a capacity-bounded model, generalizable circuits must compete with the shallow ones learned
Generalization Dynamics of LM Pre-training — Jiaxin Wen ← back Generalization Dynamics of LM Pre-training Jiaxin Wen 1 , Zhengxuan Wu 2 , Dawn Song 1 , Lijie Chen 1 1 UC Berkeley · 2 Stanford (now at Google DeepMind) May 2026 Abstract People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like modes, i.e. distinct algorithms implemented by distinct circuits. We call t
saved by
related reading
- Language model harnesses are compositional generalizersalexzhang13.github.io
- Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
- Pretraining progress is mostly coming from datadwarkesh.com
- How far does alignment midtraining generalize?alignment.openai.com
- Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
- Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
- [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
- [2602.05910] Chunky Post-Training: Data Driven Failures of Generalizationarxiv.org
- [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org
- [2606.12016] Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationarxiv.org