CausaLab — Can LLM Agents Discover Causal Mechanisms by Experiment?
A scalable environment that scores not just whether an agent gets the answer, but whether it recovered the causal mechanism — by experimenting like a scientist.
CausaLab — Can LLM Agents Discover Causal Mechanisms by Experiment? TL;DR CausaLab evaluates two things at once : did the agent solve the task, and is its answer grounded in a faithful recovered causal mechanism ? Each episode hides a freshly sampled structural causal model (SCM), so you can't win by reciting memorized causal facts. Prediction ≠ understanding. On observational 6-node graphs, GPT-5.2-high reaches 92% task accuracy but only 0.47 all-edge F 1 . The right number, the wrong graph. How you experiment is the whole game. Observation narrows the hypothesis space; agent-chosen intervent
saved by
related reading
- Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignmentarxiv.org
- Prime Agent: A Self-Improving RLM Harnessarxiv.org
- Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
- LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
- Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
- [2605.27567] Why LLMs Fail at Causal Discovery and How Interventional Agents Escapearxiv.org
- Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research] — LessWronglesswrong.com
- Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
- [2602.06337] Can Post-Training Transform LLMs into Causal Reasoners?arxiv.org
- Audit Without Verification: When LLM Accountability Layers Relay Rather Than Checkarxiv.org
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
- Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com