✳flâneur — a map of the web's best reading

Heading Toward Morality - LessWrong

lesswrong.com · 6,358 words · saved by 1 readers

Followup to: Ghosts in the Machine, Fake Fake Utility Functions, Fake Utility Functions As people were complaining before about not seeing where the quantum physics sequence was going, I shall go ahead and tell you where I'm heading now. Having dissolved the confusion surrounding the word "could", the trajectory is now heading toward should. In fact, I've been heading there for a while. Remember the whole sequence on fake utility functions? Back in... well... November 2007? I sometimes think of there being a train that goes to the Friendly AI station; but it makes several stops before it gets there; and at each stop, a large fraction of the remaining passengers get off. One of those stops is the one I spent a month leading up to in November 2007, the sequence chronicled in Fake Fake Utility Functions and concluded in Fake Utility Functions. That's the stop where someone thinks of the One Great Moral Principle That Is All We Need To Give AIs. To deliver that one warning, I had to go through all sorts of topics—which topics one might find useful even if not working on Friendly AI. I warned against Affective Death Spirals, which required recursing on the affect heuristic and halo effect, so that your good feeling about one particular moral principle wouldn't spiral out of control. I did that whole sequence on evolution ; and discursed on the human ability to make almost any goal appear to support almost any policy; I went into evolutionary psychology to argue for why we shouldn't expect human terminal values to reduce to any simple principle, even happiness, explaining the concept of "expected utility" along the way... ...and talked about genies and more; but you can read the Fake Utility sequence for that. So that's just the warning against trying to oversimplify human morality into One Great Moral Principle. If you want to actually dissolve the confusion that surrounds the word "should"—which is the next stop on the train—then that takes a much longer

x Heading Toward Morality — LessWrong Adding Up to Normality Ethics & Morality Personal Blog 28 Heading Toward Morality by Eliezer Yudkowsky 20th Jun 2008 5 min read 53 28 Followup to : Ghosts in the Machine , Fake Fake Utility Functions , Fake Utility Functions As people were complaining before about not seeing where the quantum physics sequence was going, I shall go ahead and tell you where I'm heading now. Having dissolved the confusion surrounding the word " could ", the trajectory is now heading toward should. In fact, I've been heading there for a while. Remember the whole sequence on fa

Explore this link on the map →

related reading