flâneur

Emergent Deception and Emergent Optimization

bounded-regret.ghost.io · 4,254 words · saved by 1 readers

I’ve previously argued that machine learning systems often exhibit emergent capabilities, and that these capabilities could lead to unintended negative consequences. But how can we reason concretely about these consequences?

[Note: this post was drafted before Sydney (the Bing chatbot) was released, but Sydney demonstrates some particularly good examples of some of the issues I discuss below. I've therefore added a few Sydney-related notes in relevant places.] I’ve previously argued that machine learning systems often exhibit emergent capabilities , and that these capabilities could lead to unintended negative consequences . But how can we reason concretely about these consequences? There’s two principles I find useful for reasoning about future emergent capabilities: If a capability would help get lower training

related reading