[2510.24941] Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought

Abstract:Large language models can generate long chain-of-thought (CoT) reasoning, but it remains unclear whether the verbalized steps reflect the models' internal thinking. In this work, we propose a True Thinking Score (TTS) to quantify the causal contribution of each step in CoT to the model's final prediction. Our experiments show that LLMs often interleave between true-thinking steps (which are genuinely used to compute the final output) and decorative-thinking steps (which give the appearance of reasoning but have minimal causal influence). We reveal that only a small subset of the total reasoning steps causally drive the model's prediction: e.g., on AIME, only an average of 2.3% of reasoning steps in CoT have a TTS >= 0.7 (range: 0-1) for Qwen-2.5. Furthermore, we find that LLMs can be steered to internally follow or disregard specific steps in their verbalized CoT using the identified TrueThinking direction. We highlight that self-verification steps in CoT (i.e., aha moments) can be decorative, while steering along the TrueThinking direction can force internal reasoning over these steps. Overall, our work reveals that LLMs often verbalize reasoning steps without performing them internally, challenging the efficiency of LLM reasoning and the trustworthiness of CoT.

[2510.24941] Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought --> Computer Science > Machine Learning arXiv:2510.24941 (cs) [Submitted on 28 Oct 2025 ( v1 ), last revised 26 May 2026 (this version, v4)] Title: Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought Authors: Jiachen Zhao , Yiyou Sun , Weiyan Shi , Dawn Song View a PDF of the paper titled Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought, by Jiachen Zhao and 3 other authors View PDF HTML (experimental) A

saved by

• Ishaan Panigrahi

related reading

• Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignmentarxiv.org
• [2412.06769] Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• Quantifying the Necessity of Chain of Thought through Opaque Serial Deptharxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyarxiv.org
• [2405.18915] Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersar5iv.labs.arxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• the-illusion-of-thinking.pdfml-site.cdn-apple.com
• [2510.09714] All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagearxiv.org
• Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersarxiv.org
• [2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org