Giovanni D'Antonio
giovannidantonio.com · 3,862 words · saved by 1 readers
Giovanni D'Antonio explores intelligence and human-AI collaboration at Thinking Machines Lab.
Evals are suddenly everywhere: company strategy, political decisions, advertising. Yet there is surprisingly little material explaining the science behind them. This three-part series is my attempt to explain benchmarking from first principles. We will start with the basics: Why benchmark? What question is an evaluation supposed to answer? What should we benchmark? Which tasks, users, environments, and capabilities should the data represent? How should we benchmark? How do metrics, sampling, uncertainty, and statistics turn into a score? In Part II, we will go deeper into the ways…
saved by
related reading
- Demystifying evals for AI agents \ Anthropicanthropic.com
- [1911.01547] On the Measure of Intelligencearxiv.org
- A statistical approach to model evaluations \ Anthropicanthropic.com
- Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com
- On measuring AInikilravi.substack.com
- Are AI benchmarks doomed? - by Anson Ho and Greg Burnhamepochai.substack.com
- AI Model & API Providers Analysis | Artificial Analysisartificialanalysis.ai
- Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
- Toward A Public Science of Model Behavior | Transluce AItransluce.org
- Center for Responsible, Decentralized Intelligence at Berkeleyrdi.berkeley.edu
- Why benchmarking is hard | Epoch AIepoch.ai
- Preface - The Emerging Science of Machine Learning Benchmarksmlbenchmarks.org