Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Clément Dumas4, Kit Fraser-Taliente6, Subhash Kantamneni6, Julian Minder3, Euan Ong6, Arnab Sen Sharma5, Daniel Wen1 Owain Evans2,†, Samuel Marks6,† Check off list We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity. 📄 Paper, 💻 Code, ⚙️ Demo The neural activations of large language models (LLMs) are notoriously difficult to understand. Anthropic’s mainline approach to interpreting these activations involves developing mechanistic understanding of LLM computations, for example by decomposing activations into semantically meaningful units. However, recent work has proposed an alternative, non-mechanistic approach known as LatentQA: training LLMs to directly answer question

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers Alignment Science Blog Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers Adam Karvonen 1,2 , James Chua 2 December 19, 2025 Clément Dumas 4 , Kit Fraser-Taliente 6 , Subhash Kantamneni 6 , Julian Minder 3 , Euan Ong 6 , Arnab Sen Sharma 5 , Daniel Wen 1 Owain Evans 2,† , Samuel Marks 6,† 1 MATS; 2 Truthful AI; 3 EPFL; 4 ENS Paris-Saclay; 5 Northeastern University; 6 Anthropic; † Equal advising, order randomized tl;dr We train LLMs to accept LLM neural activations as

related reading

• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• [2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersarxiv.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Natural Language Autoencoders \ Anthropicanthropic.com
• Current activation oracles are hard to use — LessWronglesswrong.com
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Transformer Circuits Threadtransformer-circuits.pub
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — LessWronglesswrong.com
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org