[2602.15029] Symmetry in language statistics shapes the geometry of model representations

Abstract:The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. These results suggest that representational manifolds have a universal origin: symmetry in the statistics of natural data.

# link_12x0urwa26e.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Dhruva Karkada; Daniel J. Korchinski; Andres Nava; Matthieu Wyart; Yasaman Bahri - Creator=arXiv GenPDF (tex2pdf:57610bf) - Custom.DOI=https://doi.org/10.48550/arXiv.2602.15029 - Custom.License=http://creativecommons.org/licenses/by-sa/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2602.15029v2 - Pr

saved by

• Sophie Wang
• Henry Zimmerman

related reading

• The Linear Representation Hypothesis and the Geometry of Large Language Modelsarxiv.org
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• The World Inside Neural Networksgoodfire.ai
• The Platonic Representation Hypothesisphillipi.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Language, trees, and geometry in neural networkspair-code.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Word Embeddingslena-voita.github.io
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• On the Biology of a Large Language Modeltransformer-circuits.pub
• Symmetry and Geometry in Neural Representationsproceedings.mlr.press
• Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu