flâneur

Flow Matching Policy Gradients

flowreinforce.github.io · 2,186 words · saved by 4 readers

Flow models have become the go-to approach to model distributions in continuous space. They soak up data with a simple, scalable denoising objective and now represent the state-of-the art in generating images, videos, audio and, more recently, robot actions. However, they’re still not widely used for learning from rewards with reinforcement learning. To perform RL in continuous spaces, practitioners typically train far simpler Gaussian policies, which represent a single, ellipsoidal mode of the action distribution. Flow-based policies can capture complex, multimodal action distributions, but they are primarily trained in a supervised manner with behavior cloning (BC). We show that it’s possible to train RL policies using flow matching, the framework behind modern diffusion and flow models, to benefit from its expressivity. We introduce Flow Policy Optimization (FPO), a new algorithm to train RL policies with flow matching. It can train expressive flow policies from only rewards. We fin

Flow Matching Policy Gradients Flow Matching Policy Gradients Simple Online Reinforcement Learning with Flow Matching David McAllister* Songwei Ge* Brent Yi* Chung Min Kim Ethan Weber Hongsuk Choi Haiwen Feng Angjoo Kanazawa Paper arXiv Code Flow models have become the go-to approach to model distributions in continuous space. They soak up data with a simple, scalable denoising objective and now represent the state-of-the art in generating images, videos, audio and, more recently, robot actions. However, they’re still not widely used for learning from rewards with reinforcement learning. To pe

saved by

related reading