[2410.06992] SWE-Bench+: Enhanced Coding Benchmark for LLMs

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which…

SWE-Bench+: Enhanced Coding Benchmark for LLMs Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, Song Wang Lassonde School of Engineering York University {reem1100,hrx00,mmm98,ennorom,guddin,wangsong}@yorku.ca Abstract Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Sev

related reading

• 2502.18449arxiv.org
• SWE-chat: Coding Agent Interactions From Real Users in the Wildarxiv.org
• Claude SWE-Bench Performance \ Anthropicanthropic.com
• [2607.00053] SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasksarxiv.org
• Senior SWE-Benchsenior-swe-bench.snorkel.ai
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Toward Training Superintelligent Software Agents through Self-Play SWE-RLarxiv.org
• Coding Models Are Doing Too Much | whnrehiew.github.io
• [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
• [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org