How much science is verifiable? Results from replicating ICML 2026 oral papers — SAI | SAI

When ICML wrapped up, we used AI agents to review and replicate every oral paper to see what we can learn from actually verifying the science.

TL;DR: AI agents provide an exciting opportunity to rethink the research ecosystem. Try our review/replication service. As AI agents become capable of increasingly complex, long-horizon tasks, we have an exciting opportunity to rethink the research ecosystem. One possibility is to verify the science rather than only reviewing the narrative in a paper, a clear blind spot in human peer review. To explore this possibility, when ICML wrapped up, we used AI agents to review and replicate every oral paper to see what we can learn from actually verifying the science. We address three questions:…

saved by

• Yixiong Hao

related reading

• The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemsarxiv.org
• Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
• ML is useful for many things, but not for predicting scientific replicabilitynormaltech.ai
• When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchalphaxiv.org
• The Need for Verification in AI-Driven Scientific Discoveryalphaxiv.org
• [2606.28277] Towards Automating Scientific Review with Google's Paper Assistant Toolarxiv.org
• What will be left for us to work on?normaltech.ai
• Can AI automate computational reproducibility?normaltech.ai
• Scientists should use AI as a tool, not an oraclenormaltech.ai
• When AI Writes the World's Software, Who Verifies It? — Leonardo de Mouraleodemoura.github.io
• Thoughts on AI in academiatheinfinitesimal.substack.com
• AI-Generated Papers in the NeurIPS 2026 Position Paper Trackblog.neurips.cc