Reevaluating Policy Gradient Methods for Imperfect-Information Games

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for four large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algori

Reevaluating Policy Gradient Methods for Imperfect-Information Games Max Rudolph University of Texas at Austin Nathan Lichtlé University of California, Berkeley Sobhan Mohammadpour Massachusetts Institute of Technology Alexandre Bayen University of California, Berkeley J. Zico Kolter Carnegie Mellon University Amy Zhang University of Texas at Austin Gabriele Farina Massachusetts Institute of Technology Eugene Vinitsky NYU Tandon School of Engineering Samuel Sokota Carnegie Mellon University Abstract In the past decade, motivated by the putative failure of naive self-play deep reinforcement lea

related reading

• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• Reinforcement Learning via Implicit Imitation Guidancearxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• 1011.0686arxiv.org
• From REINFORCE to Dr. GRPOlancelqf.github.io
• Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Playarxiv.org
• [2011.01859] A Study of Policy Gradient on a Class of Exactly Solvable Modelsarxiv.org