New Paper: Towards a science of AI agent reliability

Quantifying the capability-reliability gap

By Stephan Rabanser, Sayash Kapoor, Arvind Narayanan Suppose you hear about a new AI agent for improving productivity — by making purchases, or writing code, or sending emails, or handling a customer on your behalf. Should you trust it? Can the agent do the job reliably enough? After all, there are many horror stories of agents going wrong. Surprisingly, even though the lack of reliability of AI agents is well known, right now the AI industry doesn’t have good tools for measuring reliability, or even a good definition of reliability. Arvind and Sayash have long been thinking about this.…

related reading

• New paper: AI agents that matternormaltech.ai
• Demystifying evals for AI agents \ Anthropicanthropic.com
• Does AI Progress Have a Speed Limit?—Asteriskasteriskmag.com
• Measuring AI Ability to Complete Long Tasks - METRmetr.org
• Building reliable AI agents · parth sareenparthsareen.com
• How do we (more) safely defer to AIs? — LessWronglesswrong.com
• Measuring AI Ability to Complete Long Software Tasksarxiv.org
• Toward A Public Science of Model Behavior | Transluce AItransluce.org
• Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
• Patterns and problems in multiagent systemsanthropic.com
• More compute, more capability: Why AI agent evaluations need to account for test-time compute | AISI Workaisi.gov.uk
• Can AI automate computational reproducibility?normaltech.ai