✳flâneur — a map of the web's best reading

Niceness is unnatural — AI Alignment Forum

alignmentforum.org · 3,335 words · saved by 1 readers

When I'm arguing points like orthogonality and fragility of value, I've occasionally come across rejoinders that I'll (perhaps erroneously) summarize: Superintelligences are not spawned fully-formed; they are created by some training process. And perhaps it is in the nature of training processes, especially training processes that involve multiple agents facing "social" problems or training processes intentionally designed by humans with friendliness in mind, that the inner optimizer winds up embodying the spirit of niceness and compassion. Like, perhaps there just aren't all that many ways for a young mind to grow successfully in a world full of other agents with their own desires, and in the face of positive reinforcement for playing nicely with those agents, and negative reinforcement for crossing them. And perhaps one of the common ways for such a young mind to grow, is for it to internalize into its core goals the notions of kindness and compassion and respect-for-the-desires-of-o

x Niceness is unnatural — AI Alignment Forum 2022 MIRI Alignment Discussion AI Risk AI Frontpage 48 Niceness is unnatural by So8res 13th Oct 2022 10 min read 20 48 When I'm arguing points like orthogonality and fragility of value , I've occasionally come across rejoinders that I'll (perhaps erroneously) summarize: Superintelligences are not spawned fully-formed; they are created by some training process. And perhaps it is in the nature of training processes, especially training processes that involve multiple agents facing "social" problems or training processes intentionally designed by human

Explore this link on the map →

related reading