In (highly contingent!) defense of interpretability-in-the-loop ML training — LessWrong

Let’s call “interpretability-in-the-loop training” the idea of running a learning algorithm that involves an inscrutable trained model, and there’s s…

x In (highly contingent!) defense of interpretability-in-the-loop ML training — LessWrong Agent Foundations Interpretability (ML & AI) AI Frontpage 85 In (highly contingent!) defense of interpretability-in-the-loop ML training by Steven Byrnes 6th Feb 2026 AI Alignment Forum 4 min read 11 85 Ω 35 Let’s call “interpretability-in-the-loop training” the idea of running a learning algorithm that involves an inscrutable trained model, and there’s some kind of interpretability system feeding into the loss function / reward function. Interpretability-in-the-loop training has a very bad rap (and right

related reading

• In (highly contingent!) defense of interpretability-in-the-loop ML training — AI Alignment Forumalignmentforum.org
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• The Most Forbidden Technique — LessWronglesswrong.com
• Deep Deceptiveness — LessWronglesswrong.com
• Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover — LessWronglesswrong.com
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• On Optimism for Interpretabilitygoodfire.ai
• How To Go From Interpretability To Alignment: Just Retarget The Search — LessWronglesswrong.com
• Auditing language models for hidden objectives — LessWronglesswrong.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Assessing skeptical views of interpretability research | Christopher Pottsweb.stanford.edu