flâneur

Mesa-Optimization - AI Alignment Forum

alignmentforum.org · 611 words · saved by 1 readers

Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa-optimizer. The primary reference work for this concept is Hubinger et al.'s "Risks from Learned Optimization in Advanced Machine Learning Systems". Example: Natural selection is an optimization process that optimizes for reproductive fitness. Natural selection produced humans, who are themselves optimizers. Humans are therefore mesa-optimizers of natural selection. In the context of AI alignment, the concern is that a base optimizer (e.g., a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing.[1] Previously work under this c

x Mesa-Optimization — AI Alignment Forum Mesa-Optimization Edited by riceissa , Rob Bensinger , Ruby , et al. last updated 20th Sep 2022 Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa- optimizer . The primary reference work for this concept is Hubinger et al.'s " Risks from Learned Optimization in Advanced Machine Learning Systems ". Example: Natural selection is an optimization process that optimizes for reproductive fitness. Natural selection p

related reading