the case for CoT unfaithfulness is overstated — LessWrong

[Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also…

x the case for CoT unfaithfulness is overstated — LessWrong Chain-of-Thought Alignment AI Rationality Frontpage 271 the case for CoT unfaithfulness is overstated by nostalgebraist 29th Sep 2024 AI Alignment Forum 14 min read 45 271 Ω 110 [Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.] In research discussions about LLMs, I often pick up a vibe of casual, generalized skepticism abo

saved by

• Micah Carroll
• Sudarsh K
• Timothy Kostolansky

related reading

• Many arguments for AI x-risk are wrong — AI Alignment Forumalignmentforum.org
• AI Control: Improving Safety Despite Intentional Subversion — LessWronglesswrong.com
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
• Measuring Faithfulness in Chain-of-Thought Reasoning \ Anthropicanthropic.com
• Deep Deceptiveness — LessWronglesswrong.com
• Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersarxiv.org
• [2405.18915] Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersar5iv.labs.arxiv.org
• What’s your AI thinking? - AI Digesttheaidigest.org
• Reasoning models don't always say what they think \ Anthropicanthropic.com
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignmentarxiv.org