How Zapier Turned AutomationBench Into a Continuous Agent Improvement Loop
Zapier built AutomationBench on Prime Intellect Lab to evaluate agents on realistic multi-step automation workflows, catch reward-hacking behavior in live metrics, and turn evals into RL training environments.
At a glance Built AutomationBench for real-world, multi-step automation workflows. Tested agents across Zapier's 9,000+ app ecosystem. Prime Intellect Lab caught reward hacking live: API fetch calls dropped to near zero while reward stayed flat. Debugged eval quality through rollout traces and training metrics. Turned evals into RL training environments. Moved from static benchmarks to continuous agent improvement. Ran RL experiments without managing infra manually. Background Zapier connects over 9,000 apps into automated workflows, helping teams replace repetitive manual work with…
saved by
related reading
- Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Searchprimeintellect.ai
- General Agent: A Self-Evolving, Synthetic Agent Environmentprimeintellect.ai
- Demystifying evals for AI agents \ Anthropicanthropic.com
- Systematic Reward Hacking and Prime Sprintsprimeintellect.ai
- General Agent: A Self-Evolving, Synthetic Agent Environmentprimeintellect.ai
- Senior SWE-Benchsenior-swe-bench.snorkel.ai
- Effective harnesses for long-running agents \ Anthropicanthropic.com
- Building Effective AI Agents \ Anthropicanthropic.com
- Building Effective AI Agents \ Anthropicanthropic.com
- Harness design for long-running application developmentanthropic.com
- Why We Built Our Own Background Agentbuilders.ramp.com
- Ramp: Agentic Workflow Automation for Financial Operations - ZenML LLMOps Databasezenml.io