Current activation oracles are hard to use — LessWrong

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. • tldr; …

x Current activation oracles are hard to use — LessWrong MATS Program AI Frontpage 83 Current activation oracles are hard to use by aryaj , Senthooran Rajamanoharan , Neel Nanda 3rd Mar 2026 20 min read 4 83 This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan. tldr; Activation oracles (Karvonen et al. ) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different to what they were trained for so we decided to try t

related reading

• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2606.02609] Building Better Activation Oraclesarxiv.org
• Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWronglesswrong.com
• Alignment remains a hard, unsolved problem — LessWronglesswrong.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersalignment.anthropic.com
• Test your interpretability techniques by de-censoring Chinese models — LessWronglesswrong.com
• [2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersarxiv.org
• Current AIs seem pretty misaligned to me — LessWronglesswrong.com