[2602.15029] Symmetry in language statistics shapes the geometry of model representations

Abstract:The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. These results suggest that representational manifolds have a universal origin: symmetry in the statistics of natural data.

# link_12x0urwa26e.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Dhruva Karkada; Daniel J. Korchinski; Andres Nava; Matthieu Wyart; Yasaman Bahri - Creator=arXiv GenPDF (tex2pdf:57610bf) - Custom.DOI=https://doi.org/10.48550/arXiv.2602.15029 - Custom.License=http://creativecommons.org/licenses/by-sa/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2602.15029v2 - Pr

saved by

• Laerdon Kim
• Sophie Wang
• Henry Zimmerman

related reading

• Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• Harnessing the Universal Geometry of Embeddingsarxiv.org
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• [2602.15829] Operationalising the Superficial Alignment Hypothesis via Task Complexityarxiv.org
• [2510.26745] Deep sequence models tend to memorize geometrically; it is unclear whyarxiv.org
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelsarxiv.org
• The Linear Representation Hypothesis and the Geometry of Large Language Modelsarxiv.org
• When Models Manipulate Manifolds: The Geometry of a Counting Tasktransformer-circuits.pub
• The World Inside Neural Networksgoodfire.ai
• Language, trees, and geometry in neural networkspair-code.github.io
• 1301.3781arxiv.org
• The Geometry of Concepts: Sparse Autoencoder Feature Structurearxiv.org