Measuring what Matters: Construct Validity in Large Language Model Benchmarks | alphaXiv

Researchers systematically reviewed 445 Large Language Model benchmarks to assess construct validity, uncovering widespread issues in how abstract capabili

Understanding Construct Validity in Large Language Model Evaluation Large Language Models (LLMs) are increasingly evaluated using benchmarks that claim to measure complex capabilities like reasoning, safety, and intelligence. However, a fundamental question remains largely unexamined: do these benchmarks actually measure what they claim to measure? This paper addresses this critical gap by conducting the first large-scale systematic review of construct validity in LLM benchmarks, analyzing 445 peer-reviewed articles to understand current evaluation practices and their limitations.…

saved by

• Al-Ekram Elahee Hridoy

related reading

• Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org
• BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?alphaxiv.org
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• [2503.03750] The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• LLM evaluation: a beginner's guideevidentlyai.com
• f316275b44ee2de533102913828a8107-Paper-Datasets_and_Benchmarks_Track.pdfproceedings.neurips.cc
• Applying Statistics to LLM Evaluationscameronrwolfe.substack.com
• Foundational Challenges in Assuring Alignment and Safety of Large Language Modelsarxiv.org