Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet \ Anthropic

A post for developers about the new Claude 3.5 Sonnet and the SWE-bench eval

Our latest model, the upgraded Claude 3.5 Sonnet , achieved 49% on SWE-bench Verified, a software engineering evaluation, beating the previous state-of-the-art model's 45%. This post explains the "agent" we built around the model, and is intended to help developers get the best possible performance out of Claude 3.5 Sonnet. SWE-bench is an AI evaluation benchmark that assesses a model's ability to complete real-world software engineering tasks. Specifically, it tests how the model can resolve GitHub issues from popular open-source Python repositories. For each task in the benchmark, the AI mod

saved by

• Ratan Kaliani
• Jayaditya Sethi

related reading

• Claude Sonnet 4.5 System Cardassets.anthropic.com
• Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com
• Effective harnesses for long-running agents \ Anthropicanthropic.com
• Senior SWE-Benchsenior-swe-bench.snorkel.ai
• Best practices for Claude Code - Claude Code Docsanthropic.com
• Demystifying evals for AI agents \ Anthropicanthropic.com
• Claude Sonnet 4.5 is probably the “best coding model in the world” (at least for now)simonwillison.net
• ProgramBenchprogrambench.com
• Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations — LessWronglesswrong.com
• Claude Code Opus 4.8 Performance Tracker | Marginlabmarginlab.ai
• Automated Researchers Can Subtly Sandbagalignment.anthropic.com
• [2410.06992] SWE-Bench+: Enhanced Coding Benchmark for LLMsar5iv.labs.arxiv.org