The Platonic Representation Hypothesis

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on. Authors: achieve the best HTML results from your LaTeX submissions by following these best practices. We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get la

The Platonic Representation Hypothesis Minyoung Huh Brian Cheung Tongzhou Wang Phillip Isola Abstract We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is dri

related reading

• The Platonic Representation Hypothesisphillipi.github.io
• [2405.07987] The Platonic Representation Hypothesisarxiv.org
• All AI Models Might Be The Same - by Jack Morrisblog.jxmo.io
• The Scaling Hypothesis · Gwern.netgwern.net
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• Interpretability Dreamstransformer-circuits.pub
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• The “it” in AI models is the dataset. — Non_Intnonint.com
• On neural scaling and the quanta hypothesisericjmichaud.com
• The Linear Representation Hypothesis and the Geometry of Large Language Modelsarxiv.org
• [2205.13147] Matryoshka Representation Learningarxiv.org
• Formation of Representations in Neural Networksarxiv.org