[2207.14251] Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions

Abstract:Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. But what exactly in the training data causes a model to make a certain prediction? We seek to answer this question by providing a language for describing how training data influences predictions, through a causal framework. Importantly, our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone. Addressing the problem of extracting factual knowledge from pretrained language models (PLMs), we focus on simple data statistics such as co-occurrence counts and show that these statistics do influence the predictions of PLMs, suggesting that such models rely on shallow heuristics. Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.

Abstract:Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. But what exactly in the training data causes a model to make a certain prediction? We seek to answer this question by providing a language for describing how training data influences predictions, through a causal framework. Importantly, our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone. Addressing the problem of extracting factual knowledge from pretrained language models (PLMs), we foc

related reading

• On the Biology of a Large Language Modeltransformer-circuits.pub
• Large Language Model: world models or surface statistics?thegradient.pub
• Tracing the thoughts of a large language model \ Anthropicanthropic.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2602.06337] Can Post-Training Transform LLMs into Causal Reasoners?arxiv.org
• [2605.27567] Why LLMs Fail at Causal Discovery and How Interventional Agents Escapearxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
• Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• TRAK-ing Model Behavior with Data – gradient sciencegradientscience.org
• Pitfalls in Evaluating Language Model Forecastersarxiv.org