✳flâneur — a map of the web's best reading

Behavior Cloning is Miscalibrated - LessWrong

lesswrong.com · 1,247 words · saved by 1 readers

Behavior cloning (BC) is, put simply, when you have a bunch of human expert demonstrations and you train your policy to maximize likelihood over the human expert demonstrations. It’s the simplest possible approach under the broader umbrella of Imitation Learning, which also includes more complicated things like Inverse Reinforcement Learning or Generative Adversarial Imitation Learning. Despite its simplicity, it’s a fairly strong baseline. In fact, prompting GPT-3 to act agent-y is essentially also BC, just rather than cloning on a specific task, you're cloning against all of the task demonstration-like data in the training set--but fundamentally, it's a scaled up version of the exact same thing. The problem with BC that leads to miscalibration is that the human demonstrator may know more or less than the model, which would result in the model systematically being over/underconfident for its own knowledge and abilities. For instance, suppose the human demonstrator is more knowledgeable than the model at common sense: then, the human will ask questions about common sense much less frequently than the model should. However, with BC, the model will ask those questions at the exact same rate as the human, and then because now it has strictly less information than the human, it will have to marginalize over the possible values of the unobserved variables using its prior to be able to imitate the human’s actions. Factoring out the model’s prior over unobserved information, this is equivalent[1] to taking a guess at the remaining relevant info conditioned on all the other info it has (!!!), and then act as confidently as if it had actually observed that info, since that's how a human would act (since the human really observed that information outside of the episode, but our model has no way of knowing that). This is, needless to say, a really bad thing for safety; we want our models to ask us or otherwise seek out information whenever they don't know something, not ran

x Behavior Cloning is Miscalibrated — LessWrong Calibration Machine Learning (ML) Outer Alignment Reinforcement learning AI Frontpage 77 Behavior Cloning is Miscalibrated by leogao 5th Dec 2021 AI Alignment Forum 4 min read 3 77 Ω 36 Behavior cloning (BC) is, put simply, when you have a bunch of human expert demonstrations and you train your policy to maximize likelihood over the human expert demonstrations. It’s the simplest possible approach under the broader umbrella of Imitation Learning, which also includes more complicated things like Inverse Reinforcement Learning or Generative Adversar

Explore this link on the map →

related reading