ProgramBench
programbench.com · 4,914 words · saved by 1 readers
ProgramBench evaluates whether language models can rebuild programs from scratch.
ProgramBench ./ Program Bench Can language models rebuild programs from scratch? Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior. John Yang * , Kilian Lieret * , Jeffrey Ma , Parth Thakkar , Dmitrii Pedchenko , Sten Sootla , Emily McMilin , Pengcheng Yin , Rui Hou , Gabriel Synnaeve , Diyi Yang , Ofir Press Meta Superintelligence Labs • Stanford University • Harvard University Leaderboard Evaluated with mini-SWE-agent · 200 tasks · Updated May. 11, 2026 · S
saved by
related reading
- Senior SWE-Benchsenior-swe-bench.snorkel.ai
- Claude SWE-Bench Performance \ Anthropicanthropic.com
- Composer2.pdfcursor.com
- ReactBenchreact-bench.com
- PostTrainBenchposttrainbench.com
- Introducing FrontierCode | Cognitioncognition.com
- GitHub - openai/mle-bench: MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineeringgithub.com
- Introducing FrontierCode | Cognitioncognition.ai
- Every Benchmark is Brokenjonathanpgabor.substack.com
- Program Synthesis with Large Language Modelsarxiv.org
- Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com
- What it Takes for Coding Agents to Complete Large Software Tasksfactory.ai