[2505.12540] Harnessing the Universal Geometry of Embeddings

Abstract:We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

View PDF HTML (experimental) Abstract:We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space…

saved by

• Laerdon Kim

related reading

• Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridgearxiv.org
• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• [2510.26745] Deep sequence models tend to memorize geometrically; it is unclear whyarxiv.org
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• The Linear Representation Hypothesis and the Geometry of Large Language Modelsarxiv.org
• [2508.21038] On the Theoretical Limitations of Embedding-Based Retrievalarxiv.org
• Announcing ScaNN: Efficient Vector Similarity Searchai.googleblog.com
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• On the Theoretical Limitations of Embedding-Based Retrievalarxiv.org
• [2205.13147] Matryoshka Representation Learningarxiv.org
• [2510.01634] CAT: Curvature-Adaptive Transformers for Geometry-Aware Learningarxiv.org