The bait and switch behind AI risk prediction tools
Toronto recently used an AI tool to predict when a public beach will be safe. It went horribly awry. The developer claimed the tool achieved over 90% accuracy in predicting when beaches would be safe to swim in. But the tool did much worse: on a majority of the days when the water was in fact unsafe, beaches remained open based on the tool’s assessments. It was less accurate than the previous method of simply testing the water for bacteria each day.
Toronto recently used an AI tool to predict when a public beach will be safe. It went horribly awry. The developer claimed the tool achieved over 90% accuracy in predicting when beaches would be safe to swim in. But the tool did much worse: on a majority of the days when the water was in fact unsafe, beaches remained open based on the tool’s assessments. It was less accurate than the previous method of simply testing the water for bacteria each day. We do not find this surprising. In fact, we consider this to be the default state of affairs when an AI risk prediction tool is deployed.…
related reading
- The AI Superforecasters Are Here - by Scott Alexanderastralcodexten.com
- Frontier Risk Report (February to March 2026) - METRmetr.org
- Predicting LLM Safety Before Release by Simulating Deploymentcdn.openai.com
- Current AIs seem pretty misaligned to meblog.redwoodresearch.org
- Import AIjack-clark.net
- How fast is AI improving? - AI Digesttheaidigest.org
- near-term-xpt-accuracy.pdfstatic1.squarespace.com
- AI cannot predict the future. But companies keep trying (and failing).normaltech.ai
- My picture of the present in AI - by Ryan Greenblattblog.redwoodresearch.org
- metr.org/risk-report-feb-mar-2026.pdf#page=33.47metr.org
- AI #100: Meet the New Boss | Don't Worry About the Vasethezvi.wordpress.com
- Toward A Public Science of Model Behavior | Transluce AItransluce.org