[2109.13916] Unsolved Problems in ML Safety

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

[2109.13916] Unsolved Problems in ML Safety Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2109.13916 (cs) [Submitted on 28 Sep 2021 ( v1 ), last revised 16 Jun 2022 (this version, v5)] Title: Unsolved Problems in ML Safety Authors: Dan Hendrycks , Nicholas Carlini , John Schulman , Jacob Steinhardt View a PDF of the paper titled Unsolved Problems in ML Safety, by Dan Hendrycks and Nicholas Carlini and John Schulman and Jacob Steinhardt View PDF Abstract: Ma

related reading

• Recommendations for Technical AI Safety Research Directionsalignment.anthropic.com
• [2401.05566] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingarxiv.org
• Foundational Challenges in Assuring Alignment and Safety of Large Language Modelsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• [1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systemsarxiv.org
• [2608.07514] Open Technical Problems in Open-Weight AI Model Risk Managementarxiv.org
• [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
• [2502.14143] Multi-Agent Risks from Advanced AIarxiv.org
• [2502.05209] Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiesarxiv.org
• 2410.21680arxiv.org