A “diff” tool for AI: Finding behavioral differences in new models \ Anthropic

Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.

Interpretability A “diff” tool for AI: Finding behavioral differences in new models Mar 13, 2026 Read the paper Every time a new AI model is released, its developers run a suite of evaluations to measure its performance and safety. These tests are essential, but they are somewhat limited. Because these benchmarks are human-authored, they can only test for risks we have already conceptualized and learned to measure. This approach to safety is inherently reactive . It’s effective at catching known problems, but by definition, it's incapable of discovering “unknown unknowns”—the novel, emergent b

saved by

• Rajan Agarwal
• Devin Plumb

related reading

• Automated Alignment Researchers: Using large language models to scale scalable oversight \ Anthropicanthropic.com
• Recommendations for Technical AI Safety Research Directionsalignment.anthropic.com
• Core views on AI safety: When, why, what, and how \ Anthropicanthropic.com
• Core views on AI safety: When, why, what, and how \ Anthropicanthropic.com
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com
• Auditing language models for hidden objectives \ Anthropicanthropic.com
• A statistical approach to model evaluations \ Anthropicanthropic.com
• Insights on Crosscoder Model Diffingtransformer-circuits.pub
• AI Control: Improving Safety Despite Intentional Subversion — LessWronglesswrong.com
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org