✳flâneur — a map of the web's best reading

Part 3: Intro to Policy Optimization — Spinning Up documentation

spinningup.openai.com · 3,255 words · saved by 5 readers

In this section, we’ll discuss the mathematical foundations of policy optimization algorithms, and connect the material to sample code. We will cover three key results in the theory of policy gradients: In the end, we’ll tie those results together and describe the advantage-based expression for the policy gradient—the version we use in our Vanilla Policy Gradient implementation. Here, we consider the case of a stochastic, parameterized policy, . We aim to maximize the expected return . For the purposes of this derivation, we’ll take to give the finite-horizon undiscounted return, but the derivation for the infinite-horizon discounted return setting is almost identical. We would like to optimize the policy by gradient ascent, eg The gradient of policy performance, , is called the policy gradient, and algorithms that optimize the policy this way are called policy gradient algorithms. (Examples include Vanilla Policy Gradient and TRPO. PPO is often referred to as a policy gradient algori

Part 3: Intro to Policy Optimization - Spinning Up documentation --> Docs >> Part 3: Intro to Policy Optimization Edit on GitHub Part 3: Intro to Policy Optimization ¶ Table of Contents Part 3: Intro to Policy Optimization Deriving the Simplest Policy Gradient Implementing the Simplest Policy Gradient Expected Grad-Log-Prob Lemma Don’t Let the Past Distract You Implementing Reward-to-Go Policy Gradient Baselines in Policy Gradients Other Forms of the Policy Gradient Recap In this section, we’ll discuss the mathematical foundations of policy optimization algorithms, and connect the

Explore this link on the map →

saved by

related reading