Proximal Policy Optimization Algorithms | alphaXiv
alphaXiv 0 CY V2 CLAIM AUTHORSHIP Proximal Policy Optimization Algorithms John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov OpenAI { joschu, filip, prafulla, alec, oleg } @openai.com Abstract We propose a new family of policy gradient methods for reinforcement learning, which al- ternate between sampling data through interaction with the environment, and optimizing a “surrogate” objective function using stochastic gradient ascent. Whereas standard policy gra- dient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimiza- tion (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, includ- ing simulated robotic locom
Problem Statement and Motivation Reinforcement learning (RL) has made significant progress in recent years, but developing algorithms that are simultaneously data-efficient, robust, and simple to implement remains challenging. Policy gradient methods, while conceptually straightforward, often suffer from poor sample efficiency and instability when multiple updates are performed on the same batch of data. Trust Region Policy Optimization (TRPO) addressed these issues by constraining policy updates to maintain stability, but its implementation complexity—requiring second-order optimization…
saved by
related reading
- [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
- The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
- [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- Policy Gradient Algorithms | Lil'Loglilianweng.github.io
- Proximal Policy Optimization (PPO)huggingface.co
- Policy gradient methoden.wikipedia.org
- Proximal Policy Optimization - Spinning Up documentationspinningup.openai.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- Understanding Policy Gradients | John Lambertjohnwlambert.github.io
- RLHF Bookrlhfbook.com
- Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io