Why do Policy Gradient Methods work so well in Cooperative MARL? Evidence from Policy Representation – The Berkeley Artificial Intelligence Research Blog

The BAIR Blog

In cooperative multi-agent reinforcement learning (MARL), due to its on-policy nature, policy gradient (PG) methods are typically believed to be less sample efficient than value decomposition (VD) methods, which are off-policy . However, some recent empirical studies demonstrate that with proper input representation and hyper-parameter tuning, multi-agent PG can achieve surprisingly strong performance compared to off-policy VD methods. Why could PG methods work so well? In this post, we will present concrete analysis to show that in certain scenarios, e.g., environments with a highly multi-mod

related reading

• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• MARLlib: A Multi-agent Reinforcement Learning Library — MARLlib v1.0.0 documentationmarllib.readthedocs.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• [2206.12765] Generalized Beliefs for Cooperative AIarxiv.org
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Reevaluating Policy Gradient Methods for Imperfect-Information Gamesarxiv.org
• JaxMARL: Multi-Agent RL, but 10000x Fasterblog.foersterlab.com