Single Token Geometry: DeepSeek V4 and Manifold Tearing

单标几何：DeepSeek V4 与流形撕裂

Chinese Edition: 单标几何 02: 流形撕裂 I was preparing Single Token Geometry: Data Complexity, the second entry in this series , when DeepSeek V4 dropped. I set it aside immediately. What caught my attention wasn’t the benchmark numbers, impressive as they are. It was a paragraph buried in Section 4.2.3 of the technical report: “We identified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers.” And then, with unusual candor: “Although a comprehensive theoretical understanding of…

saved by

• Sam B
• Arya Pasumarthi

related reading

• [2602.15799] The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safetyarxiv.org
• The Local Learning Coefficient: A Singularity-Aware Complexity Measurearxiv.org
• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• When Models Manipulate Manifolds: The Geometry of a Counting Tasktransformer-circuits.pub
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• The World Inside Neural Networksgoodfire.ai
• [2510.01634] CAT: Curvature-Adaptive Transformers for Geometry-Aware Learningarxiv.org
• Uncovering Neural Geometry in Vision Models With Block-Sparse Featurizersgoodfire.ai
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• [2512.24880] mHC: Manifold-Constrained Hyper-Connectionsarxiv.org