Proximal Policy Optimization — Spinning Up documentation

PPO is motivated by the same question as TRPO: how can we take the biggest possible improvement step on a policy using the data we currently have, without stepping so far that we accidentally cause performance collapse? Where TRPO tries to solve this problem with a complex second-order method, PPO is a family of first-order methods that use a few other tricks to keep new policies close to old. PPO methods are significantly simpler to implement, and empirically seem to perform at least as well as TRPO. There are two primary variants of PPO: PPO-Penalty and PPO-Clip. PPO-Penalty approximately solves a KL-constrained update like TRPO, but penalizes the KL-divergence in the objective function instead of making it a hard constraint, and automatically adjusts the penalty coefficient over the course of training so that it’s scaled appropriately. PPO-Clip doesn’t have a KL-divergence term in the objective and doesn’t have a constraint at all. Instead relies on specialized clipping in the objec

Proximal Policy Optimization - Spinning Up documentation --> Docs >> Proximal Policy Optimization Edit on GitHub Proximal Policy Optimization ¶ Table of Contents Proximal Policy Optimization Background Quick Facts Key Equations Exploration vs. Exploitation Pseudocode Documentation Documentation: PyTorch Version Saved Model Contents: PyTorch Version Documentation: Tensorflow Version Saved Model Contents: Tensorflow Version References Relevant Papers Why These Papers? Other Public Implementations Background ¶ (Previously: Background for TRPO ) PPO is motivated by the same question as TRPO: how c

related reading

• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• A Graphic Guide to Implementing PPO for Atari Games | Towards Data Sciencetowardsdatascience.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• Vanilla Policy Gradient - Spinning Up documentationspinningup.openai.com
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• RLHF Bookrlhfbook.com