Worries about latent reasoning in LLMs — EA Forum

When working through a problem, OpenAI's o1 model will write a chain-of-thought (CoT) in English. This CoT reasoning is human-interpretable by defaul…

SummaryBot 1y 1 0 0 Executive summary: The post discusses the emerging paradigm of latent reasoning in large language models (LLMs) like COCONUT, which offers a potentially more efficient but less interpretable alternative to traditional chain-of-thought (CoT) reasoning. Key points: The COCONUT model uses a continuous latent space for reasoning, abandoning the human-readable chain-of-thought for a vector-based approach that encodes multiple reasoning paths simultaneously. This method shows promise in specific logical reasoning tasks by reducing the number of forward passes needed compared to C

related reading

• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• [2412.06769] Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• the case for CoT unfaithfulness is overstated — LessWronglesswrong.com
• The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
• Chain-of-Thought Promptinglearnprompting.org
• [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyarxiv.org
• Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
• Learning to reason with LLMs | OpenAIopenai.com
• Unlocking the Working Memory of Large Language Models for Latent Reasoningarxiv.org
• How will we do SFT on models with opaque reasoning? — AI Alignment Forumalignmentforum.org
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org