[2605.27567] Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Abstract:Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, \emph{not any particular model or dataset}. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing

Abstract:Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal represen

related reading

• CausaLab — Can LLM Agents Discover Causal Mechanisms by Experiment?dylanzsz.github.io
• [2602.06337] Can Post-Training Transform LLMs into Causal Reasoners?arxiv.org
• Tracing the thoughts of a large language model \ Anthropicanthropic.com
• Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
• Large Language Model: world models or surface statistics?thegradient.pub
• Reasoning Models Reason Well, Until They Don'tarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Thought Branches: Interpreting LLM Reasoning Requires Resamplingarxiv.org
• Reasoning as Trajectoriesslhleosun.github.io
• Bryon Aragam // University of Chicagobryonaragam.com
• Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasonersarxiv.org
• Frederick Eberhardtits.caltech.edu