Towards a Typology of Strange LLM Chains-of-Thought — LessWrong

LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of speculation about why. But a lot of this speculation narrows too quickly, to just one or two hypotheses. My intent is also to speculate, but more broadly. Specifically, I want to outline six nonexclusive possible causes for the weird tokens: new better language, spandrels, context refresh, deliberate obfuscation, natural drift, and conflicting shards. And I also wish to extremely roughly outline ideas for experiments and evidence that could help us distinguish these causes. I'm sure I'm not enumerating the full space of possibilities. I'm also sure that I'm probably making some mistakes in what follows, or confusing my

x Towards a Typology of Strange LLM Chains-of-Thought — LessWrong Language Models (LLMs) AI Curated 2025 Top Fifty: 12 % 311 Towards a Typology of Strange LLM Chains-of-Thought by 1a3orn 9th Oct 2025 10 min read 29 311 Intro LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of s

related reading

• Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• LLM Daydreaming · Gwern.netgwern.net
• Chain of Draft: Thinking Faster by Writing Lessarxiv.org
• [2510.24941] Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thoughtarxiv.org
• [2412.06769] Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• the case for CoT unfaithfulness is overstated — LessWronglesswrong.com
• Vestigial reasoning in RL — LessWronglesswrong.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org