LLMs Are Closing the Gap on Human Superforecasters

We opened our AI forecasting benchmark to external submissions. Here’s what happened.

(Post written by Houtan Bastani, Simas Kučinskas, and Matt Reynolds)In October, we opened our AI forecasting benchmark, ForecastBench, to external submissions. The challenge: beat superforecasters using any tools available, from scaffolding to fine-tuning. Several teams responded, including xAI, Cassi, Lightning Rod, and Mantic. We thank all of them for participating on this challenging benchmark. The result? External submissions now hold #2 and #3 on our leaderboard, outperforming all our baseline LLM configurations. Superforecasters still hold #1. Here are the current standings (lower…

saved by

• Rohan Kanti

related reading

• The AI Superforecasters Are Here - by Scott Alexanderastralcodexten.com
• Training LLMs to Predict World Events (Guest Post with Mantic) - Thinking Machines Labthinkingmachines.ai
• Exploreforecastbench.org
• 2025 LLM Year in Review – karpathykarpathy.bearblog.dev
• Pitfalls in Evaluating Language Model Forecastersarxiv.org
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Successful language model evals - Jason Weijasonwei.net
• PostTrainBenchposttrainbench.com
• GitHub - openai/mle-bench: MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineeringgithub.com
• [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
• How well did forecasters predict 2025 AI progress? | Epoch AIepoch.ai
• Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org