The Crux of Every AI System: Evaluations | WHOOP Engineering
engineering.prod.whoop.com · 1,776 words · saved by 1 readers
How we built a framework from scratch to ensure quality and stability for WHOOP AI.
Across the industry, we see AI features shipped on hope alone. At WHOOP, we ship with data and security in mind. In order to support over 500 unique agents, we built an evaluation framework that treats LLMs like the statistical, noisy systems they are. Here's exactly how it works. The Problem We've built AI Studio to enable anyone at WHOOP to develop and interact with our homegrown Agents, resulting in an explosion of more than 500 of them across virtually every screen in the app. But as we reduced the friction to build Agents, the new bottleneck became testing them. Manual dogfooding turns
related reading
- Demystifying evals for AI agents \ Anthropicanthropic.com
- Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
- Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
- AI agent evaluation frameworks for production - Vercelvercel.com
- A pragmatic guide to LLM evals for devsnewsletter.pragmaticengineer.com
- Agentic Evals Pyramidrwilinski.ai
- From Idea To Agent In Less Than Ten Minutes | WHOOP Engineeringengineering.prod.whoop.com
- Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
- Challenges in evaluating AI systems \ Anthropicanthropic.com
- Building Effective AI Agents \ Anthropicanthropic.com