flâneur

Vanilla Policy Gradient — Spinning Up documentation

spinningup.openai.com · 1,406 words · saved by 2 readers

The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return, until you arrive at the optimal policy. Let denote a policy with parameters , and denote the expected finite-horizon undiscounted return of the policy. The gradient of is where is a trajectory and is the advantage function for the current policy. The policy gradient algorithm works by updating policy parameters via stochastic gradient ascent on policy performance: Policy gradient implementations typically compute advantage function estimates based on the infinite-horizon discounted return, despite otherwise using the finite-horizon undiscounted policy gradient formula. VPG trains a stochastic policy in an on-policy way. This means that it explores by sampling actions according to the latest version of its stochastic policy. The amount of randomness in action selection depends on both initial conditions

Vanilla Policy Gradient - Spinning Up documentation --> Docs >> Vanilla Policy Gradient Edit on GitHub Vanilla Policy Gradient ¶ Table of Contents Vanilla Policy Gradient Background Quick Facts Key Equations Exploration vs. Exploitation Pseudocode Documentation Documentation: PyTorch Version Saved Model Contents: PyTorch Version Documentation: Tensorflow Version Saved Model Contents: Tensorflow Version References Relevant Papers Why These Papers? Other Public Implementations Background ¶ (Previously: Introduction to RL, Part 3 ) The key idea underlying policy gradients is to push up the probab

saved by

related reading