Risk-Averse AIs
We argue that training AIs to be risk-averse – to treat resources as having diminishing marginal utility – could both preserve AIs’ usefulness (if they turn out aligned) and provide an extra line of defense (if they turn out misaligned).
23rd June 2026 [AI Narration] Risk-Averse AIs Playback speed Volume 0:00 of 1:56:03 Abstract We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest…
saved by
related reading
- Many arguments for AI x-risk are wrong — AI Alignment Forumalignmentforum.org
- AI Deterrence by Betrayalaibetrayal.com
- Why are AI agents lying, cheating and coordinating?yoshuabengio.org
- The case for ensuring that powerful AIs are controlled — LessWronglesswrong.com
- Can risk aversion learned at low stakes generalize to astronomically high stakes?substack.com
- Fail safe(r) at alignment by channeling reward-hacking into a "spillway" motivationblog.redwoodresearch.org
- How do we (more) safely defer to AIs? — LessWronglesswrong.com
- Risk from fitness-seeking AIs: mechanisms and mitigationssubstack.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Thoughts on the conservative assumptions in AI controlblog.redwoodresearch.org
- Making deals with early schemers — LessWronglesswrong.com
- The Case Against AI Control Research — LessWronglesswrong.com