Can AI automate computational reproducibility?

A new benchmark to measure the impact of AI on improving science

Last month, Sakana AI released an "AI scientist", which the company called "the first comprehensive system for fully automatic scientific discovery". It was touted as being able to accelerate science without suffering from human limitations. Unfortunately, the "AI Scientist" has many shortcomings. It has no checks for novelty, so generated papers could rehash earlier work. And Sakana did not perform any human review (let alone expert “peer” review) of the generated papers—so it is unclear if the papers are any good (apparently they are not). While these flaws are particularly flagrant in…

related reading

• Are AI benchmarks doomed? - by Anson Ho and Greg Burnhamepochai.substack.com
• Measuring AI Ability to Complete Long Software Tasksarxiv.org
• How much science is verifiable? Results from replicating ICML 2026 oral paperssai.science
• When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchalphaxiv.org
• Center for Responsible, Decentralized Intelligence at Berkeleyrdi.berkeley.edu
• Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentsalphaxiv.org
• Import AI 455: AI systems are about to start building themselves.importai.substack.com
• Toward A Public Science of Model Behavior | Transluce AItransluce.org
• f316275b44ee2de533102913828a8107-Paper-Datasets_and_Benchmarks_Track.pdfproceedings.neurips.cc
• The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemsarxiv.org
• On measuring AInikilravi.substack.com
• New paper: AI agents that matternormaltech.ai