✳flâneur — a map of the web's best reading

Instrumental convergence — LessWrong

lesswrong.com · 5,836 words · saved by 1 readers

Alternative introductions * Steve Omohundro: "The Basic AI Drives" * Nick Bostrom: "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents". Introduction: A machine of unknown purpose Suppose you landed on a distant planet and found a structure of giant metal pipes, crossed by occasional cables. Further investigation shows that the cables are electrical superconductors carrying high-voltage currents. You might not know what the huge structure did. But you would nonetheless guess that this huge structure had been built by some intelligence, rather than being a naturally-occurring mineral formation - that there were aliens who built the structure for some purpose. Your reasoning might go something like this: "Well, I don't know if the aliens were trying to manufacture cars, or build computers, or what. But if you consider the problem of efficient manufacturing, it might involve mining resources in one place and then efficiently transporting them somewhere else, like by pipes. Since the most efficient size and location of these pipes would be stable, you'd want the shape of the pipes to be stable, which you could do by making the pipes out of a hard material like metal. There's all sorts of operations that require energy or negentropy, and a superconducting cable carrying electricity seems like an efficient way of transporting that energy. So I don't know what the aliens were ultimately trying to do, but across a very wide range of possible goals, an intelligent alien might want to build a superconducting cable to pursue that goal." That is: We can take an enormous variety of compactly specifiable goals, like "travel to the other side of the universe" or "support biological life" or "make paperclips", and find very similar optimal strategies along the way. Today we don't actually know if electrical superconductors are the most useful way to transport energy in the limit of technology. But whatever is the most efficient w

x Instrumental convergence — LessWrong Main 6 LW Wiki 1 Instrumental convergence Edited by Eliezer Yudkowsky , et al. last updated 19th Feb 2025 Alternative introductions Steve Omohundro: " The Basic AI Drives " Nick Bostrom: " The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents ". Introduction: A machine of unknown purpose Suppose you landed on a distant planet and found a structure of giant metal pipes, crossed by occasional cables. Further investigation shows that the cables are electrical superconductors carrying high-voltage currents. You might

Explore this link on the map →

saved by

related reading