Introducing the REFORMS checklist for ML-based science

ML-based science is in trouble. Clear reporting standards for researchers could help.

See the REFORMS pre-print and checklist here. Every music producer is looking for the next hit song. So when a paper claimed that machine learning can predict hit songs with 97% accuracy, it would have been music to their ears. News outlets, including Scientific American and Axios, published pieces about how this "frightening accuracy" could revolutionize the music industry. Earlier studies have found that it is hard to predict if a song will be successful in advance, so this paper seemed to be a dramatic achievement. Unfortunately for music producers, we found that the study's results are…

related reading

• The ML drug discovery startup trying really, really hard to not cheat (Leash Bio)owlposting.com
• Scientists should use AI as a tool, not an oraclenormaltech.ai
• Sloppy Use of Machine Learning Is Causing a ‘Reproducibility Crisis’ in Science | WIREDwired.com
• [2108.02497] How to avoid machine learning pitfalls: a guide for academic researchersarxiv.org
• ML is useful for many things, but not for predicting scientific replicabilitynormaltech.ai
• Preface - The Emerging Science of Machine Learning Benchmarksmlbenchmarks.org
• The speed of science - Works in Progress Magazineworksinprogress.co
• Five things to keep in mind while reading biology ML papersowlposting.com
• Context Widowsartificialbureaucracy.substack.com
• Why are Machine Learning Projects so Hard to Manage? | by Lukas Biewald | Mediummedium.com
• Questionable practices in machine learningarxiv.org
• My Criteria for Reviewing Papersblog.evjang.com