Textbook | AIMS
Decisions about AI systems—whether to deploy one, which to prefer, how to govern its use—rest on measurements of quantities such as reasoning, planning, and safety. The quality of those measurements is therefore a precondition for sound decisions, yet it is rarely examined. New benchmarks appear faster than their measurement properties can be characterized; leaderboards report scores without specifying the scale on which they are defined; and a system’s average accuracy is largely uninformative about why it succeeds or where it fails. The field has accumulated a large number of evaluation results, but comparatively little theory governing how those results should be produced and interpreted. This book develops the foundations and practical tools of measurement science for AI evaluation. Its organizing premise is that evaluation is a problem of inference - where we estimate latent properties of a system, such as ability along a given dimension, from observed responses, - rather than a p