Measuring what Matters: Construct Validity in Large Language Model Benchmarks | alphaXiv

View recent discussion. Abstract: Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

Understanding Construct Validity in Large Language Model Evaluation Large Language Models (LLMs) are increasingly evaluated using benchmarks that claim to measure complex capabilities like reasoning, safety, and intelligence. However, a fundamental question remains largely unexamined: do these benchmarks actually measure what they claim to measure? This paper addresses this critical gap by conducting the first large-scale systematic review of construct validity in LLM benchmarks, analyzing 445 peer-reviewed articles to understand current evaluation practices and their limitations.…

saved by

• Al-Ekram Elahee Hridoy

related reading

• Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org
• BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?alphaxiv.org
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• LLM evaluation: a beginner's guideevidentlyai.com
• [2503.03750] The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemsarxiv.org
• Applying Statistics to LLM Evaluationscameronrwolfe.substack.com
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
• Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com