Formal verification, heuristic explanations and surprise accounting — Alignment Research Center

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language. To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives. In this post, we will: We are also sharing a draft by Gabriel Wu (c

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language. To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechani

saved by

• Asher P

related reading

• ARC progress update: Competing with sampling — LessWronglesswrong.com
• AlgZoo: uninterpreted models with fewer than 1,500 parameters — LessWronglesswrong.com
• A gentle introduction to mechanistic anomaly detection — LessWronglesswrong.com
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
• Toy Models of Superpositiontransformer-circuits.pub
• Competing with sampling — Alignment Research Centeralignment.org
• An Ambitious Vision for Interpretability — AI Alignment Forumalignmentforum.org
• A Mike's-Eye View of ARC's Research — Alignment Research Centeralignment.org
• Research update: Towards a Law of Iterated Expectations for Heuristic Estimators — Alignment Research Centeralignment.org
• Neural networks are not already cleanlesswrong.com