Towards a Typology of Strange LLM Chains-of-Thought

LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of speculation about why. But a lot of this speculation narrows too quickly, to just one or two hypotheses. My intent is also to speculate, but more broadly. Specifically, I want to outline six nonexclusive possible causes for the weird tokens: new better language, spandrels, context refresh, deliberate obfuscation, natural drift, and conflicting shards. And I also wish to outline ideas for experiments and evidence that could help us distinguish these causes. I'm sure I'm not enumerating the full space of possibilities. I'm also sure that I'm probably making some mistakes what follows, or confusing my ontologies. But it's

Towards a Typology of Strange LLM Chains-of-Thought 1 A 3 O R N 1 A 3 O R N Towards a Typology of Strange LLM Chains-of-Thought On the Causes of LLM Unintelligibility Created: 2025-10-02 Wordcount: 2k Tags: philosophy effort-post machine-learning empiricism Intro LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why

related reading

• Towards a Typology of Strange LLM Chains-of-Thought — LessWronglesswrong.com
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• LLM Daydreaming · Gwern.netgwern.net
• llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• Thought Branches: Interpreting LLM Reasoning Requires Resamplingarxiv.org
• [2405.18915] Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersar5iv.labs.arxiv.org
• [2510.27338] Reasoning Models Sometimes Output Illegible Chains of Thoughtarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2510.24941] Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thoughtarxiv.org
• Vestigial reasoning in RL — LessWronglesswrong.com
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com