ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering | alphaXiv

ALE-Bench provides the first benchmark for long-duration, score-based algorithmic programming contests, co-developed with AtCoder to evaluate AI systems on complex optimization problems. It...

Abstract How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over

saved by

• Ziran Yang

related reading

• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale | alphaXivalphaxiv.org
• LLM-as-a-Verifier: A General-Purpose Verification Framework | alphaXivalphaxiv.org
• EdgeBench | Scaling Laws of Environment Learningedge-bench.org
• [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
• AI Model & API Providers Analysis | Artificial Analysisartificialanalysis.ai
• GitHub - openai/mle-bench: MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineeringgithub.com
• [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
• ProgramBenchprogrambench.com
• AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms — Google DeepMinddeepmind.google
• PostTrainBenchposttrainbench.com
• Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com
• Measuring AI Ability to Complete Long Software Tasksarxiv.org