Bayesianism versus conservatism versus Goodhart — AI Alignment Forum

Key argument: if we use a non-Bayesian conservative approach, such as a minimum over different utility functions, then we better have a good reason a…

x Bayesianism versus conservatism versus Goodhart — AI Alignment Forum Goodhart's Law AI Frontpage 8 Bayesianism versus conservatism versus Goodhart by Stuart_Armstrong 16th Jul 2021 7 min read 2 8 Key argument: if we use a non-Bayesian conservative approach, such as a minimum over different utility functions, then we better have a good reason as to why that would work. But if we have that reason, we can use it to make the whole thing into a Bayesian mix, which can also allow us to trade off that advantage against other possible gains. I've defended using Bayesian averaging of possible utility

related reading

• Too much efficiency makes everything worse: overfitting and the strong version of Goodhart’s law | Jascha’s blogsohl-dickstein.github.io
• Bayesian Mindsetcold-takes.com
• Approximately Bayesian Reasoning: Knightian Uncertainty, Goodhart, and the Look-Elsewhere Effect — LessWronglesswrong.com
• Why I'm not a Bayesianmindthefuture.info
• Thoughts on the conservative assumptions in AI controlblog.redwoodresearch.org
• Efficient tradeoffs and the safety-usefulness tradeoff model — LessWronglesswrong.com
• Goodhart's Law — AI Alignment Forumalignmentforum.org
• A positive case for how we might succeed at prosaic AI alignment — AI Alignment Forumalignmentforum.org
• Scaling Laws for Reward Model Overoptimizationarxiv.org
• Yudhister Kumaryudhister.me
• Probabilities, Prioritization, and Bayesian Mindsettheviolethour.substack.com
• The Bayes Banditfrancesco215.github.io