✳flâneur — a map of the web's best reading

Vingean uncertainty — LessWrong

lesswrong.com · 1,596 words · saved by 1 readers

> Of course, I never wrote the “important” story, the sequel about the first amplified human. Once I tried something similar. John Campbell’s letter of rejection began: “Sorry—you can’t write this story. Neither can anyone else.”... “Bookworm, Run!” and its lesson were important to me. Here I had tried a straightforward extrapolation of technology, and found myself precipitated over an abyss. It’s a problem writers face every time we consider the creation of intelligences greater than our own. When this happens, human history will have reached a kind of singularity—a place where extrapolation breaks down and new models must be applied—and the world will pass beyond our understanding. -- Vernor Vinge, True Names and other Dangers, p. 47. Vingean unpredictability is a key part of how we think about a consequentialist intelligence which we believe is smarter than us in a domain. In particular, we usually think we can't predict exactly what a smarter-than-us agent will do, because if we could predict that, we would be that smart ourselves (Vinge's Principle). If you could predict exactly what action Deep Blue would take on a chessboard, you could play as well as Deep Blue by making whatever move you predicted Deep Blue would make. It follows that Deep Blue's programmers necessarily sacrificed their ability to intuit Deep Blue's exact moves in advance, in the course of creating a superhuman chessplayer. But this doesn't mean Deep Blue's programmers were confused about the criterion by which Deep Blue chose actions. Deep Blue's programmers still knew in advance that Deep Blue would try to win rather than lose chess games. They knew that Deep Blue would try to steer the chess board's future into a particular region that was high in Deep Blue's preference ordering over chess positions. We can predict the consequences of Deep Blue's moves better than we can predict the moves themselves. "Vingean uncertainty" is the peculiar epistemic state we enter when we're considering

x Vingean uncertainty — LessWrong Vingean uncertainty Edited by Eliezer Yudkowsky , et al. last updated 21st Jun 2016 Of course, I never wrote the “important” story, the sequel about the first amplified human. Once I tried something similar. John Campbell’s letter of rejection began: “Sorry—you can’t write this story. Neither can anyone else.”... “Bookworm, Run!” and its lesson were important to me. Here I had tried a straightforward extrapolation of technology, and found myself precipitated over an abyss. It’s a problem writers face every time we consider the creation of intelligences greater

Explore this link on the map →

related reading