RIP Classic Reasoning Benchmarks. What’s Next?
epochai.substack.com · 1,683 words · saved by 1 readers
Give up at least one of: text only, short time horizon, easy to grade, and expert human superiority.
Gradient Updates RIP Classic Reasoning Benchmarks. What’s Next? Give up at least one of: text only, short time horizon, easy to grade, and expert human superiority. Greg Burnham May 05, 2026 40 2 3 Share This post is part of Epoch AI’s Gradient Updates newsletter, which shares more opinionated or informal takes on big questions in AI progress. These posts solely represent the views of the authors, and do not necessarily reflect the views of Epoch AI as a whole. Originally posted on Epoch AI . There’s a familiar recipe for reasoning benchmarks: tasks are text-only, output is easy to grade, and
related reading
- Are AI benchmarks doomed? - by Anson Ho and Greg Burnhamepochai.substack.com
- [1911.01547] On the Measure of Intelligencearxiv.org
- AI in 2025: gestalt — LessWronglesswrong.com
- AI progress is about to speed up | Epoch AIepoch.ai
- My picture of the present in AI — LessWronglesswrong.com
- How close is AI to taking my job? | Epoch AIepoch.ai
- Toward an O*NET for AI R&Depochai.substack.com
- After Automation | Everyevery.to
- The case for multi-decade AI timelinesepochai.substack.com
- Clarifying and predicting AGI — LessWronglesswrong.com
- If AGI Means Everything People Do... What is it That People Do?secondthoughts.ai
- Why benchmarking is hard | Epoch AIepoch.ai