Actor-Critic Algorithms

We propose and analyze a class of actor-critic algorithms for simulation-based optimization of a Markov decision process over a parameterized family of randomized stationary policies. These are two-time-scale algorithms in which the critic uses TD learning with a linear approximation architecture and the actor is updated in an approximate gradient direction based on information pro(cid:173) vided by the critic. We show that the features for the critic should span a subspace prescribed by the choice of parameterization of the actor. We conclude by discussing convergence properties and some open problems.

Vijay R. Konda, John N. Tsitsiklis Advances in Neural Information Processing Systems 12 (NIPS 1999) Abstract We propose and analyze a class of actor-critic algorithms for simulation-based optimization of a Markov decision process over a parameterized family of randomized stationary policies. These are two-time-scale algorithms in which the critic uses TD learning with a linear approximation architecture and the actor is updated in an approximate gradient direction based on information pro(cid:173) vided by the critic. We show that the features for the critic should span a subspace…

saved by

• Marco Jiralerspong

related reading

• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• RLAlgsInMDPs.pdfsites.ualberta.ca
• Soft Actor-Critic - Spinning Up documentationspinningup.openai.com
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• The idea behind Actor-Critics and how A2C and A3C improve them | AI Summertheaisummer.com
• 1512.04455arxiv.org
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• [1801.01290] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorarxiv.org
• Soft Actor-Critic for continuous and discrete actions | by Wah Loon Keng | Mediummedium.com
• Policy gradient methoden.wikipedia.org
• [2011.01859] A Study of Policy Gradient on a Class of Exactly Solvable Modelsarxiv.org
• RL ALGOk-a.in