The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity - Apple Machine Learning Research
Authors Parshin Shojaee*†, Iman Mirzadeh*, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar View publication Copy Bibtex Recent generations of frontier language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established mathematical and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from data contamination and does not provide insights into the reasoning traces’ structure and quality. In this work, we systematically investigate these gaps with the help of controllable puzzle environments that allow precise manipulation of compositional complexity while maintaining consistent logical structures. This setup enables the analysis of
AuthorsParshin Shojaee*†, Iman Mirzadeh*, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar Recent generations of frontier language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established mathematical and coding benchmarks, emphasizing final answer accuracy. However, this evaluation…
saved by
related reading
- the-illusion-of-thinking.pdfml-site.cdn-apple.com
- [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
- [2412.06769] Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
- Reasoning Models Reason Well, Until They Don'tarxiv.org
- Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
- [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
- Thinking to recall: How reasoning unlocks parametric knowledge in LLMsresearch.google
- Base Models Know How to Reason, Thinking Models Learn Whenarxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- 2410.05229arxiv.org
- Unlocking the Working Memory of Large Language Models for Latent Reasoningarxiv.org
- Quantifying the Necessity of Chain of Thought through Opaque Serial Deptharxiv.org