Current activation oracles are hard to use — LessWrong

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. • tldr; …

x Current activation oracles are hard to use — LessWrong MATS Program AI Frontpage 83 Current activation oracles are hard to use by aryaj , Senthooran Rajamanoharan , Neel Nanda 3rd Mar 2026 20 min read 4 83 This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. tldr; Activation oracles (Karvonen et al. ) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different to what they were trained for so we decided to try t

related reading

• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2606.02609] Building Better Activation Oraclesarxiv.org
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersalignment.anthropic.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWronglesswrong.com
• Natural Language Autoencoders \ Anthropicanthropic.com
• [2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersarxiv.org
• GitHub - ceselder/cot-oracle: Our improvements to activation oracles · GitHubgithub.com
• cot-oracle/AObench at main · ceselder/cot-oracle · GitHubgithub.com