Towards a Typology of Strange LLM Chains-of-Thought — LessWrong

LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of speculation about why. But a lot of this speculation narrows too quickly, to just one or two hypotheses. My intent is also to speculate, but more broadly. Specifically, I want to outline six nonexclusive possible causes for the weird tokens: new better language, spandrels, context refresh, deliberate obfuscation, natural drift, and conflicting shards. And I also wish to extremely roughly outline ideas for experiments and evidence that could help us distinguish these causes. I'm sure I'm not enumerating the full space of possibilities. I'm also sure that I'm probably making some mistakes in what follows, or confusing my

x Towards a Typology of Strange LLM Chains-of-Thought — LessWrong Language Models (LLMs) AI Curated 2025 Top Fifty: 12 % 311 Towards a Typology of Strange LLM Chains-of-Thought by 1a3orn 9th Oct 2025 10 min read 29 311 Intro LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of s

related reading

• Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
• LLM Daydreaming · Gwern.netgwern.net
• [2510.24941] Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thoughtarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Vestigial reasoning in RL — LessWronglesswrong.com
• [2510.27338] Reasoning Models Sometimes Output Illegible Chains of Thoughtarxiv.org
• [2405.18915] Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersar5iv.labs.arxiv.org
• Thought Anchors: Which LLM Reasoning Steps Matter? — LessWronglesswrong.com
• Thought Branches: Interpreting LLM Reasoning Requires Resamplingarxiv.org
• the case for CoT unfaithfulness is overstated — LessWronglesswrong.com