Optimizing The Final Output Can Obfuscate CoT (Research Note) — LessWrong

Produced as part of MATS 8.0 under the mentorship of Alex Turner and Alex Cloud. This research note overviews some early results which we are looking for feedback on. TL;DR: We train language models with RL in toy environments. We show that penalizing some property of the output is sufficient to suppress that property in the chain of thought also, even when that property is relevant to task completion. For example, when we penalize a model for mentioning in its output that it completed a task via a certain form of cheating, its reasoning also omits this fact. We therefore suspect that training against an output monitor is sufficient to cause obfuscated CoTs in more realistic settings. Chain of thought (CoT) supervision appears in many control and scalable oversight protocols. It has been argued that being able to monitor CoTs for unwanted behavior is a critical property for safety that we should seek to maintain. Recent work by OpenAI showed that using a CoT monitor as a training signa

x [Research Note] Optimizing The Final Output Can Obfuscate CoT — LessWrong MATS Program Chain-of-Thought Alignment Scalable Oversight AI Frontpage 2025 Top Fifty: 14 % 202 [Research Note] Optimizing The Final Output Can Obfuscate CoT by lukemarks , jacob_drori , cloud , TurnTrout 30th Jul 2025 AI Alignment Forum 7 min read 23 202 Ω 78 Produced as part of MATS 8.0 under the mentorship of Alex Turner and Alex Cloud. This research note overviews some early results which we are looking for feedback on. TL;DR: We train language models with RL in toy environments. We show that penalizing some prope

related reading

• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
• When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorsarxiv.org
• Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilityarxiv.org
• The Most Forbidden Technique — LessWronglesswrong.com
• the case for CoT unfaithfulness is overstated — LessWronglesswrong.com
• Paper: Prompt Optimization Makes Misalignment Legible — LessWronglesswrong.com
• [2603.05706] Reasoning Models Struggle to Control their Chains of Thoughtarxiv.org
• How confessions can keep language models honest | OpenAIopenai.com
• Training Qwen-1.5B with a CoT legibility penalty — LessWronglesswrong.com
• The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
• Vestigial reasoning in RL — LessWronglesswrong.com
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com