Joshua Achiam on X: "A very hot take: chain of thought interpretability was always going to be so fragile as to be an unacceptable backstop for long-term AI safety, and while I admire the optimism and effort involved in protecting its fidelity (and consider such effort to have been worthwhile), I do" / X
A very hot take: chain of thought interpretability was always going to be so fragile as to be an unacceptable backstop for long-term AI safety, and while I admire the optimism and effort involved in protecting its fidelity (and consider such effort to have been worthwhile), I do not think it makes …
A very hot take: chain of thought interpretability was always going to be so fragile as to be an unacceptable backstop for long-term AI safety, and while I admire the optimism and effort involved in protecting its fidelity (and consider such effort to have been worthwhile), I do not think it makes sense to elevate as a principle the idea that the chain of thought must remain legible to humans. I would go so far as to say that strategies predicated on that principle are definitely doomed, in that they will not work eventually, and we should not depend on them or take enduring reassurance from…
saved by
related reading
- The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
- Dario Amodei — The Urgency of Interpretabilitydarioamodei.com
- [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyarxiv.org
- Why Frontier AI Labs Fight to Hide Chain of Thought — Ilia Shumailov & Alexander Panfilovyoutube.com
- The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
- How AI Is Learning to Think in Secretnickandresen.substack.com
- the case for CoT unfaithfulness is overstated — LessWronglesswrong.com
- The Most Forbidden Technique — LessWronglesswrong.com
- On Optimism for Interpretabilitygoodfire.ai
- An Alien Mind: Jakub Pachocki Warns Usthezvi.substack.com
- Legible vs. Illegible AI Safety Problems — LessWronglesswrong.com
- How Can Interpretability Researchers Help AGI Go Well? — AI Alignment Forumalignmentforum.org