[1912.02875] Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions

Abstract:We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [63] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also also conceptually simplify an approach [60] for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.

View PDF HTML (experimental) Abstract:We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high…

saved by

• Emanuel Ruzak

related reading

• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Life lessons from reinforcement learning - Jason Weijasonwei.net
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• RUDDER - Reinforcement Learning with Delayed Rewards | rudderml-jku.github.io
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• Reinforcement learning - Wikipediaen.wikipedia.org
• The Promise of Hierarchical Reinforcement Learningthegradient.pub
• Should I Use Offline RL or Imitation Learning? – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu
• Ch. 21 - Imitation Learningunderactuated.mit.edu
• If it makes you feel any better, I've been doing this for a while and it took me... | Hacker Newsnews.ycombinator.com
• Successor Features for Transfer in Reinforcement Learningproceedings.neurips.cc