[2606.02609] Building Better Activation Oracles

Abstract:Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make them hard to evaluate. To this end, we improve the Activation Oracle (AO) training regime in four ways: training on on-policy rollouts, improving the conversational dataset, feeding more layers and an improvement to the injection formula. The capability improvements are marginal, but quality of life improvements are quite substantial. In addition, we open source the first comprehensive evaluation suite for AO quality, which we call AObench. Overall, we hope that our work sets a foundation that helps improve AOs and other models in the paradigm of scalable, end-to-end interpretability.

Building Better Activation Oracles Jan Bauer∗ Celeste De Schamphelaere∗ Adam Karvonen MATS MATS Independent Gatsby Unit, UCL Ghent University Niclas Luick Neel Nanda MATS,…

related reading

• [2606.02609] Building Better Activation Oraclesarxiv.org
• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• Current activation oracles are hard to use — LessWronglesswrong.com
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersalignment.anthropic.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• [2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersarxiv.org
• Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWronglesswrong.com
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• GitHub - ceselder/cot-oracle: Our improvements to activation oracles · GitHubgithub.com
• Natural Language Autoencoders \ Anthropicanthropic.com