Towards a Typology of Strange LLM Chains-of-Thought

1a3orn.com · 2,609 words · saved by 5 readers

LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why might this happen? I've seen a lot of speculation about why. But a lot of this speculation narrows too quickly, to just one or two hypotheses. My intent is also to speculate, but more broadly. Specifically, I want to outline six nonexclusive possible causes for the weird tokens: new better language, spandrels, context refresh, deliberate obfuscation, natural drift, and conflicting shards. And I also wish to outline ideas for experiments and evidence that could help us distinguish these causes. I'm sure I'm not enumerating the full space of possibilities. I'm also sure that I'm probably making some mistakes what follows, or confusing my ontologies. But it's

Towards a Typology of Strange LLM Chains-of-Thought 1 A 3 O R N 1 A 3 O R N Towards a Typology of Strange LLM Chains-of-Thought On the Causes of LLM Unintelligibility Created: 2025-10-02 Wordcount: 2k Tags: philosophy effort-post machine-learning empiricism Intro LLMs being trained with RLVR (Reinforcement Learning from Verifiable Rewards) start off with a 'chain-of-thought' (CoT) in whatever language the LLM was originally trained on. But after a long period of training, the CoT sometimes starts to look very weird; to resemble no human language; or even to grow completely unintelligible. Why

Towards a Typology of Strange LLM Chains-of-Thought

saved by

related reading