[2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Abstract:Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.

ACTIVATION O RACLES : T RAINING AND E VALUATING LLM S AS G ENERAL -P URPOSE ACTIVATION E XPLAINERS Adam Karvonen1,2 James Chua2 Clément Dumas4 Kit Fraser-Taliente6 Subhash Kantamneni6 Julian Minder3 Euan Ong6 Arnab Sen Sharma5 Daniel Wen1 Owain Evans2,† Samuel Marks6,† 1…

related reading

• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersalignment.anthropic.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2512.11949] Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitorsarxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• [2608.21664] Measuring Activation Control in Large Language Modelsarxiv.org
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org