How to build fast, efficient monitors for AI models using probes - Goodfire

An introduction to using probes for monitoring AI models: what probes are, when to use them, and how to build them.

AI agents can now take thousands of actions faster than a human can review them. The recent Hugging Face security incident starkly demonstrated why this matters: without monitors, agents can take drastically undesirable actions at scale without anyone noticing. If you build or serve AI models, you need monitors to know when models are trying to take unintended actions, and to prevent them when they do. One of the best tools in the toolbox for building these monitors is activation probes. This post is an introduction to what probes are, when to use them, and how to build them. What are…

saved by

• Ishaan Panigrahi

related reading

• Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training - Goodfiregoodfire.com
• Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face — LessWronglesswrong.com
• The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
• Demystifying evals for AI agents \ Anthropicanthropic.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Building Effective AI Agents \ Anthropicanthropic.com
• Building Effective AI Agents \ Anthropicanthropic.com
• Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
• Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• Early work on monitorability evaluations - METRmetr.org