[2310.06824] The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

Abstract:Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

Abstract:Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizati

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• How well do truth probes generalise? — LessWronglesswrong.com
• Large Language Model: world models or surface statistics?thegradient.pub
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Believe It or Not: How Deeply do LLMs Believe Implanted Facts?alignment.anthropic.com
• Standards for Belief Representations in LLMsarxiv.org
• BIG-bench/bigbench/benchmark_tasks/convinceme at main · google/BIG-bench · GitHubgithub.com
• Reasoning as Trajectoriesslhleosun.github.io
• Do large language models understand us? | by Blaise Aguera y Arcas | Mediummedium.com
• Do language models possess knowledge (soundness)? - HackMDhackmd.io
• Cognitive Biases in Large Language Models — LessWronglesswrong.com
• Actually, Othello-GPT Has A Linear Emergent World Representation - Neel Nandaneelnanda.io