[2605.27567] Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Abstract:Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, \emph{not any particular model or dataset}. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing

W HY LLM S FAIL AT C AUSAL D ISCOVERY AND H OW I NTERVENTIONAL AGENTS E SCAPE Amartya Roy Sonali Parbhoo SIRE, IIT Delhi and Robert Bosch GmbH, India Imperial College London srz248670@iitd.ac.in s.parbhoo@imperial.ac.uk A BSTRACT…

related reading

• CausaLab — Can LLM Agents Discover Causal Mechanisms by Experiment?dylanzsz.github.io
• As Rocks May Think | Eric Jangevjang.com
• [2602.06337] Can Post-Training Transform LLMs into Causal Reasoners?arxiv.org
• Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
• Large Language Model: world models or surface statistics?thegradient.pub
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Is AI Reasoning Right for the Wrong Reasons? | Quanta Magazinequantamagazine.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
• Reasoning Models Reason Well, Until They Don'tarxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• Bryon Aragam // University of Chicagobryonaragam.com
• Thought Branches: Interpreting LLM Reasoning Requires Resamplingarxiv.org