Evaluating LLMs is a minefield

Annotated slides from a recent talk

We have released annotated slides for a talk titled Evaluating LLMs is a minefield. We show that current ways of evaluating chatbots and large language models don't work well, especially for questions about their societal impact. There are no quick fixes, and research is needed to improve evaluation methods. The challenges we highlight are somewhat distinct from those faced by builders of LLMs or by developers interested in comparing between LLMs for adoption. Those challenges are better understood and tackled by evaluation frameworks such as HELM. You can view the annotated slides here.…

saved by

• Myke Walton

related reading

• Challenges in evaluating AI systems \ Anthropicanthropic.com
• Evaluating LLMs is a minefieldnormaltech.ai
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Noam Brown on X: "Implications of Large-Scale Test-Time Compute" / Xx.com
• Successful language model evals - Jason Weijasonwei.net
• The bitter lesson of LLM evalsparsed.com
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• Thoughts on AI in academiatheinfinitesimal.substack.com
• Pitfalls in Evaluating Language Model Forecastersarxiv.org
• LLM evaluation: a beginner's guideevidentlyai.com
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org