The idea behind Actor-Critics and how A2C and A3C improve them | AI Summer

Actor critics, A2C, A3C

It’s time for some Reinforcement Learning. This time our main topic is Actor-Critic algorithms, which are the base behind almost every modern RL method from Proximal Policy Optimization to A3C. So, to understand all those new techniques, you should have a good grasp of what Actor-Critic are and how they work. But don’t be in a hurry. Let’s refresh for a moment on our previous knowledge. As you may know, there are two main types of RL methods out there: Value Based: They try to find or approximate the optimal value function, which is a mapping between an action and a value. The higher the value

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Soft Actor-Critic - Spinning Up documentationspinningup.openai.com
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• pdfopenreview.net
• Algorithms — Ray 2.55.1docs.ray.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org