Senior SWE-Bench

Evaluating agents as senior engineers on the work we actually give them

Senior SWE-Bench Senior SWE-Bench We treat agents like senior engineers, so why evaluate them like junior engineers? 01 Senior engineers build features without over-specified requirements Senior SWE-Bench feature tasks have realistic instructions that read like natural language messages rather than over-specified requirements. To reliably evaluate these tasks, we introduce a validation agent which uses expert-designed recipes to write behavioral tests that adapt to submitted solutions. 02 Senior engineers solve bugs that require runtime investigation from behavioral reports Senior SWE-Bench bu

saved by

• Emma Guo
• Maanav Khaitan

related reading

• Senior SWE-Benchsenior-swe-bench.snorkel.ai
• Unlocking Real-Time Bug Detection at Cognition | Applied Computeappliedcompute.com
• Killing Coding Agent Slop With Adversarial Self-Playusetelos.ai
• Agentic Evals Pyramidrwilinski.ai
• Training an Agentic Router for Optimal Cost-Performance on SWE Tasks | Applied Computeappliedcompute.com
• Notes on the Software Factorybenedict.dev
• Demystifying evals for AI agents \ Anthropicanthropic.com
• Claude SWE-Bench Performance \ Anthropicanthropic.com
• Why We Built Our Own Background Agentbuilders.ramp.com
• SWE-chat: Coding Agent Interactions From Real Users in the Wildarxiv.org
• Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Searchprimeintellect.ai
• Toward Training Superintelligent Software Agents through Self-Play SWE-RLarxiv.org