f316275b44ee2de533102913828a8107-Paper-Datasets_and_Benchmarks_Track.pdf

N/A

Establishing Best Practices for Building Rigorous Agentic Benchmarks Yuxuan Zhu1∗ Tengjun Jin1 Yada Pruksachatkun Andy Zhang2 Shu Liu3 Sasha Cui4 Sayash Kapoor5 Shayne Longpre6 Kevin Meng7 Rebecca Weiss8 Fazl Barez8,11 Rahul Gupta9 Jwala Dhamala9 Jacob Merizian10 Mario Giulianelli10 Harry Coppock10 Cozmin Ududec10 Jasjeet Sekhon4 Jacob Steinhardt7 Antony Kellermann1 Sarah Schwettmann7 Matei Zaharia3 Ion Stoica3 Percy Liang2 Daniel Kang1∗ 1 2 UIUC…

saved by

• Anton Otaner

related reading

• Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
• Center for Responsible, Decentralized Intelligence at Berkeleyrdi.berkeley.edu
• Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com
• Measuring AI Ability to Complete Long Software Tasksarxiv.org
• Research Areas in Benchmark Design and Evaluation (The Alignment Project by UK AISI) — AI Alignment Forumalignmentforum.org
• [2602.12316] GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theoryarxiv.org
• [2606.05405] Agents' Last Examarxiv.org
• ROGUE:arxiv.org
• [2602.22755] AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviorsarxiv.org
• Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org
• New paper: AI agents that matternormaltech.ai
• Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org