[2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
Abstract:Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.
# link_1oieezxorpa.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20240627004217Z - Creator=LaTeX with hyperref - ModDate=D:20240627004217Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5 - Producer=pdfTeX-1.40.25 - Trapped=False ## Contents ### Page 1 Published as a conference paper at ICLR 2023EMERGENT WORLD REPRESENTATIONS: EXPLORING A SEQUENCE MODEL TRAINED ON A SYNTHETIC TASKKenneth Li∗Harvar
saved by
related reading
- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
- Actually, Othello-GPT Has A Linear Emergent World Representation - Neel Nandaneelnanda.io
- Structure and Interpretation of Deep Networkssidn.baulab.info
- Large Language Model: world models or surface statistics?thegradient.pub
- Actually, Othello-GPT Has A Linear Emergent World Representation — AI Alignment Forumalignmentforum.org
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- Chess-GPT’s Internal World Model | Adam Karvonenadamkarvonen.github.io
- Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
- Training Language Models to Explain Their Own Computationsarxiv.org
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- The Linear Representation Hypothesis and the Geometry of Large Language Modelsarxiv.org
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org