The Stamp Collector
Once upon a time, a group of naïve philosophers found a robot that collected trinkets. Well, more specifically, the robot seemed to collect stamps: if you presented this robot with a choice between various trinkets, it would always choose the option that led towards it having as many stamps as
Once upon a time, a group of naïve philosophers found a robot that collected trinkets. Well, more specifically, the robot seemed to collect stamps: if you presented this robot with a choice between various trinkets, it would always choose the option that led towards it having as many stamps as possible in its inventory. It ignored dice, bottle caps, aluminum cans, sticks, twigs, and so on, except insofar as it predicted they could be traded for stamps in the next turn or two. So, of course, the philosophers started calling it the "stamp collector." Then, one day, the philosophers discovered co
saved by
related reading
- The Stamp Collector — LessWronglesswrong.com
- Embedded Agency (full-text version) — LessWronglesswrong.com
- Optimality is the tiger, and agents are its teeth — LessWronglesswrong.com
- Orthogonality Thesis — LessWronglesswrong.com
- Deep Deceptiveness — LessWronglesswrong.com
- Towards a scale-free theory of intelligent agencymindthefuture.info
- [1902.09469] Embedded Agencyarxiv.org
- Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
- Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org
- Models Don't "Get Reward" — LessWronglesswrong.com
- musings and rough draftsmusingsandroughdrafts.com
- Robot Philosophysubstack.com