[2505.12540] Harnessing the Universal Geometry of Embeddings

Abstract:We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

View PDF HTML (experimental) Abstract:We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space…

saved by

• Laerdon Kim

related reading

• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• Embeddings: What they are and why they mattersimonwillison.net
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Vector embeddings | OpenAI APIdevelopers.openai.com
• Vector embeddings | OpenAI APIplatform.openai.com
• Introducing text and code embeddings | OpenAIopenai.com
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Introduction to Text Embeddingstxt.cohere.com
• Meet AI’s multitool: Vector embeddings | Google Cloud Blogcloud.google.com
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• Word Embeddingslena-voita.github.io