Can AI automate computational reproducibility?

A new benchmark to measure the impact of AI on improving science

Last month, Sakana AI released an "AI scientist", which the company called "the first comprehensive system for fully automatic scientific discovery". It was touted as being able to accelerate science without suffering from human limitations. Unfortunately, the "AI Scientist" has many shortcomings. It has no checks for novelty, so generated papers could rehash earlier work. And Sakana did not perform any human review (let alone expert “peer” review) of the generated papers—so it is unclear if the papers are any good (apparently they are not). While these flaws are particularly flagrant in…

related reading

• The machines are fine. I'm worried about us.ergosphere.blog
• Danger, AI Scientist, Danger - by Zvi Mowshowitzthezvi.substack.com
• Are AI benchmarks doomed? - by Anson Ho and Greg Burnhamepochai.substack.com
• How much science is verifiable? Results from replicating ICML 2026 oral paperssai.science
• Meet the Humans Building AI Scientists - Asimov Pressasimov.press
• Import AI 455: AI systems are about to start building themselves.importai.substack.com
• Import AIjack-clark.net
• [2606.28277] Towards Automating Scientific Review with Google's Paper Assistant Toolarxiv.org
• The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemsarxiv.org
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverysakana.ai
• Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentsalphaxiv.org
• Can AI Solve Science?-Stephen Wolfram Writingswritings.stephenwolfram.com