High-level visual representations in the human brain are aligned with large language models | Nature Machine Intelligence

Doerig, Kietzmann and colleagues show that the brain’s response to visual scenes can be modelled using language-based AI representations. By linking brain activity to caption-based embeddings from large language models, the study reveals a way to quantify complex visual understanding.

Download PDF Subjects Cognitive neuroscience Neural encoding A preprint version of the article is available at arXiv. Abstract The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of

saved by

• Kira Tang

related reading

• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• On the Biology of a Large Language Modeltransformer-circuits.pub
• Do large language models understand us? | by Blaise Aguera y Arcas | Mediummedium.com
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
• Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
• Language Models in Plato's Cave - by Sergey Levinesergeylevine.substack.com
• Emotion Concepts and their Function in a Large Language Modeltransformer-circuits.pub
• LLM Daydreaming · Gwern.netgwern.net
• Large Language Model: world models or surface statistics?thegradient.pub
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com
• Emotion concepts and their function in a large language model \ Anthropicanthropic.com
• Jean-Rémi KINGkingjr.github.io