How much science is verifiable? Results from replicating ICML 2026 oral papers — SAI | SAI
sai.science · 1,473 words · saved by 1 readers
When ICML wrapped up, we used AI agents to review and replicate every oral paper to see what we can learn from actually verifying the science.
TL;DR: AI agents provide an exciting opportunity to rethink the research ecosystem. Try our review/replication service. As AI agents become capable of increasingly complex, long-horizon tasks, we have an exciting opportunity to rethink the research ecosystem. One possibility is to verify the science rather than only reviewing the narrative in a paper, a clear blind spot in human peer review. To explore this possibility, when ICML wrapped up, we used AI agents to review and replicate every oral paper to see what we can learn from actually verifying the science. We address three questions:…
saved by
related reading
- The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemsarxiv.org
- [2606.28277] Towards Automating Scientific Review with Google's Paper Assistant Toolarxiv.org
- The machines are fine. I'm worried about us.ergosphere.blog
- Can AI automate computational reproducibility?normaltech.ai
- AI-Generated Papers in the NeurIPS 2026 Position Paper Trackblog.neurips.cc
- Demystifying evals for AI agents \ Anthropicanthropic.com
- To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysisarxiv.org
- When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchalphaxiv.org
- Call for Papers | AI for Meta-Science @ NeurIPS 2026ai4metascience.org
- ML is useful for many things, but not for predicting scientific replicabilitynormaltech.ai
- pdfopenreview.net
- Reviewer Instructions 2026icml.cc