Contrastive Learning as Goal-Conditioned Reinforcement Learning
In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e.g., auxiliary losses, data augmentation). How can we design RL algorithms that directly acquire good representations? In this paper, instead of adding representation learning parts to an existing RL algorithm, we show (contrastive) representation learning methods can be cast as RL algorithms in their own right. To do this, we build upon prior work and apply contrastive representation learning to action-labeled trajectories, in such a way that the (inner product of) learned representations exactly corresponds to a goal-conditioned value function. We use this idea to reinterpret a prior RL method as performing contrastive learning, and then use the idea to pro
Tianjun Zhang Affiliation: UC Berkeley Sergey Levine Affiliation: Google Research Affiliation: UC Berkeley Ruslan Salakhutdinov Affiliation: CMU Abstract In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e.g., auxiliary losses, data augmentation). How can we design RL algorithms that directly acquire good…
saved by
related reading
- Decoupling Representation Learning from Reinforcement Learningarxiv.org
- A Free Lunch from the Noise:Provable and Practical Exploration for Representation Learningarxiv.org
- A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
- [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
- [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
- Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Researcharxiv.org
- Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
- Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
- [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org
- 2006.10742arxiv.org
- [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
- Reinforcement Learning via Implicit Imitation Guidancearxiv.org