✳flâneur — a map of the web's best reading

[1810.08575] Supervising strong learners by amplifying weak experts

ar5iv.labs.arxiv.org · 9,386 words · saved by 1 readers

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017b), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors. If we want to train an ML system to perform a task, we need to be able to evaluate how well it is doing. Whether our training signal takes the form of labels, rewards, or somethi

Supervising strong learners by amplifying weak experts Paul Christiano OpenAI paul@openai.com &Buck Shlegeris bshlegeris@gmail.com &Dario Amodei OpenAI damodei@openai.com Work done while at OpenAI. Abstract Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an

Explore this link on the map →

related reading