The bitter lesson of LLM evals

Supercharge, interpret and evolve your AI systems.

The bitter lesson of LLM evals Position July 13, 2025 The bitter lesson of LLM evals Turning expert judgment into a compounding moat. Because in LLM evals, scaling care beats scaling compute. Authors Affiliations Charles O'Neill Parsed Mudith Jayasekara Parsed Max Kirkby Parsed In the history of artificial intelligence, there is a famous principle known as the Bitter Lesson. Coined by Rich Sutton, it observes that the biggest gains in performance have consistently come not from elaborate, human-designed knowledge, but from general methods that leverage massive increases in computation. The les

saved by

• Ratan Kaliani
• Akira Yoshiyama

related reading

• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• Lil'Loglilianweng.github.io
• Does the Bitter Lesson Have Limits?dbreunig.com
• The Bitter Lesson is Misunderstoodobviouslywrong.substack.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Animals vs Ghosts – karpathykarpathy.bearblog.dev
• Will scaling work?dwarkeshpatel.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev