Pitfalls in Evaluating Language Model Forecasters

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation performance to real-world forecasting. Through systematic analysis and concrete examples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidentl

Pitfalls in Evaluating Language Model Forecasters Daniel Paleka 1,∗ Shashwat Goel 2,3,∗ Jonas Geiping 2,3 Florian Tramèr 1 1 ETH Zurich 2 ELLIS Institute Tübingen 3 MPI Tübingen Abstract Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of te

saved by

• Daniel Paleka

related reading

• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Mostarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• Training LLMs to Predict World Events (Guest Post with Mantic) - Thinking Machines Labthinkingmachines.ai
• [2607.14345] Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Valuesarxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org