Introducing Supabase Evals
supabase.com · 1,108 words · saved by 1 readers
Our open-source benchmark for how well AI coding agents build with Supabase.
Today we're open sourcing supabase/evals, our benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real Supabase tasks, for example, building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, and then scores how well they performed. It powers both our published benchmark and an internal regression suite we monitor daily. As more people ship Supabase projects through an agent instead of by hand, we wanted a way to measure that experience instead of guessing. The results…
saved by
related reading
- Demystifying evals for AI agents \ Anthropicanthropic.com
- Why We Built Our Own Background Agentbuilders.ramp.com
- A Guide to Claude Code 2.0 and getting better at using coding agents – sankalp's blogsankalp.bearblog.dev
- Agent Leaderboards · Which tools coding agents choose · Armaturearmature.tech
- AI agent evaluation frameworks for production - Vercelvercel.com
- Agentationagentation.dev
- Senior SWE-Benchsenior-swe-bench.snorkel.ai
- Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebasedatabricks.com
- Cookbookcookbook.openai.com
- Building Effective AI Agents \ Anthropicanthropic.com
- Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
- Agentationagentation.com