flâneur

Optimizing The Final Output Can Obfuscate CoT (Research Note) — LessWrong

lesswrong.com · 5,314 words · saved by 1 readers

Produced as part of MATS 8.0 under the mentorship of Alex Turner and Alex Cloud. This research note overviews some early results which we are looking for feedback on. TL;DR: We train language models with RL in toy environments. We show that penalizing some property of the output is sufficient to suppress that property in the chain of thought also, even when that property is relevant to task completion. For example, when we penalize a model for mentioning in its output that it completed a task via a certain form of cheating, its reasoning also omits this fact. We therefore suspect that training against an output monitor is sufficient to cause obfuscated CoTs in more realistic settings. Chain of thought (CoT) supervision appears in many control and scalable oversight protocols. It has been argued that being able to monitor CoTs for unwanted behavior is a critical property for safety that we should seek to maintain. Recent work by OpenAI showed that using a CoT monitor as a training signa

x [Research Note] Optimizing The Final Output Can Obfuscate CoT — LessWrong MATS Program Chain-of-Thought Alignment Scalable Oversight AI Frontpage 2025 Top Fifty: 14 % 202 [Research Note] Optimizing The Final Output Can Obfuscate CoT by lukemarks , jacob_drori , cloud , TurnTrout 30th Jul 2025 AI Alignment Forum 7 min read 23 202 Ω 78 Produced as part of MATS 8.0 under the mentorship of Alex Turner and Alex Cloud. This research note overviews some early results which we are looking for feedback on. TL;DR: We train language models with RL in toy environments. We show that penalizing some prope

related reading