Single Token Geometry: DeepSeek V4 and Manifold Tearing

单标几何：DeepSeek V4 与流形撕裂

Chinese Edition: 单标几何 02: 流形撕裂 I was preparing Single Token Geometry: Data Complexity, the second entry in this series , when DeepSeek V4 dropped. I set it aside immediately. What caught my attention wasn’t the benchmark numbers, impressive as they are. It was a paragraph buried in Section 4.2.3 of the technical report: “We identified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers.” And then, with unusual candor: “Although a comprehensive theoretical understanding of…

saved by

• Sam B
• Arya Pasumarthi

related reading

• [2602.15799] The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safetyarxiv.org
• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• When Models Manipulate Manifolds: The Geometry of a Counting Tasktransformer-circuits.pub
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• The World Inside Neural Networksgoodfire.ai
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugesarxiv.org
• arxiv.org/pdf/2512.24880#page=3.56arxiv.org
• David Pfaudavidpfau.com
• DeepSeek and the Day Before New Year'swheremachinesthink.substack.com
• Loss Landscape | A.I deep learning explorations of morphology & dynamicslosslandscape.com
• [2510.26745] Deep sequence models tend to memorize geometrically; it is unclear whyarxiv.org