[2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

Abstract:Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations Evan Miller Anthropic evanmiller@anthropic.com arXiv:2411.00640v1 [stat.AP] 1 Nov 2024 November 4, 2024…

saved by

• Akira Yoshiyama
• Evie Hu

related reading

• Applying Statistics to LLM Evaluationscameronrwolfe.substack.com
• A statistical approach to model evaluations \ Anthropicanthropic.com
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• [2608.21664] Measuring Activation Control in Large Language Modelsarxiv.org
• [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Pitfalls in Evaluating Language Model Forecastersarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• [2502.05209] Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiesarxiv.org