Introducing the REFORMS checklist for ML-based science

ML-based science is in trouble. Clear reporting standards for researchers could help.

See the REFORMS pre-print and checklist here. Every music producer is looking for the next hit song. So when a paper claimed that machine learning can predict hit songs with 97% accuracy, it would have been music to their ears. News outlets, including Scientific American and Axios, published pieces about how this "frightening accuracy" could revolutionize the music industry. Earlier studies have found that it is hard to predict if a song will be successful in advance, so this paper seemed to be a dramatic achievement. Unfortunately for music producers, we found that the study's results are…

related reading

• Scientists should use AI as a tool, not an oraclenormaltech.ai
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemsarxiv.org
• The speed of science - Works in Progress Magazineworksinprogress.co
• ML is useful for many things, but not for predicting scientific replicabilitynormaltech.ai
• Five things to keep in mind while reading biology ML papersowlposting.com
• Questionable practices in machine learningarxiv.org
• [2108.02497] How to avoid machine learning pitfalls: a guide for academic researchersarxiv.org
• How much science is verifiable? Results from replicating ICML 2026 oral paperssai.science
• Pitfalls in Evaluating Language Model Forecastersarxiv.org
• Preface - The Emerging Science of Machine Learning Benchmarksmlbenchmarks.org
• When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchalphaxiv.org