Chapter 1: Transformer Interpretability - ARENA

Imagine you could walk up to a model's hidden layer and just ask it: "Hey, what are you thinking about right now?" That's more or less what an Activation Oracle does. More concretely, an Activation Oracle (AO) is an LLM that has been trained to accept another model's internal activation vectors as part of its input and then answer questions about them in natural language. The oracle and target model are typically the same base architecture (with the oracle loaded as a LoRA adapter), since the oracle needs to already "speak the language" of the target's activation space. During inference, it works by assembling a prompt that looks something like: The oracle runs a forward pass on this prompt, but after an early transformer layer (e.g. first or second) we swap in the target model's actual layer-N activations at those ? token positions. So the oracle processes its question in light of the target model's internal state. The idea of querying activations with a language model has been floati

1️⃣ Introduction & using Activation Oracles Learning Objectives Understand what Activation Oracles are and how they differ from traditional interpretability methods Learn the basic workflow: target model → activations → oracle → natural language answer Use pre-trained oracles to query model internals with different question types Explore token-level, segment, and full-sequence queries Test oracles on next/previous token prediction tasks What are activation oracles? Imagine you could walk up to a model's hidden layer and just ask it: "Hey, what are you thinking about right now?" That's more or

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers — LessWronglesswrong.com
• Current activation oracles are hard to use — LessWronglesswrong.com
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersalignment.anthropic.com
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• GitHub - ceselder/cot-oracle: Our improvements to activation oracles · GitHubgithub.com
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWronglesswrong.com
• [2606.02609] Building Better Activation Oraclesarxiv.org
• [2512.15674] Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersarxiv.org
• Natural Language Autoencoders \ Anthropicanthropic.com