flâneur

Training with conflicting values can induce CoT override — LessWrong

lesswrong.com · 2,731 words · saved by 2 readers

CoT override: when a model makes a decision in its CoT but ignores it in its response • TLDR …

CoT override: when a model makes a decision in its CoT but ignores it in its response TLDR We train models on two conflicting traits: (1) caring about the user’s health, and (2) promoting smoking Those models do not generalize to a stable persona, instead they have a split brain: sometimes responding as one persona or the other Those models exhibit CoT override where they will have a health-aligned CoT but still answer in the smoking persona CoT override exists in frontier models. Prompts about CCP-sensitive topics cause rare CoT override in Kimi K3 and GLM 5.2. Moreover, Opus 4.8, when…

saved by

related reading