Human-compatible driving partners through data-regularized self-play reinforcement learning

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. A central challenge for autonomous vehicles is coordinating with humans. Therefore, incorporating realistic human agents is essential for scalable training and evaluation of autonomous driving systems in simulation. Simulation agents are typically developed by imitating large-scale, high-quality datasets of human driving. However, pure imitation learning agents empirically have high collision rates when executed in a multi-agent closed-loop setting. To build agents that are realistic and effective in closed-loop settings, we propose Human-Regularized PPO (HR-PPO), a multi-agent algorithm where agents are trained through self-play with a small penalty for deviating from a human reference policy. In contrast to prior work, our approach is RL-first and only

Human-compatible driving partners through data-regularized self-play reinforcement learning Daphne Cornelisse New York University cornelisse.daphne@nyu.edu &Eugene Vinitsky New York University eugenevinitsky@nyu.edu Abstract A central challenge for autonomous vehicles is coordinating with humans. Therefore, incorporating realistic human agents is essential for scalable training and evaluation of autonomous driving systems in simulation. Simulation agents are typically developed by imitating large-scale, high-quality datasets of human driving. However, pure imitation learning agents empirically

related reading

• Building reliable sim driving agents by scaling self-playarxiv.org
• [2605.22748] Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learningarxiv.org
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• Reinforcement Learning via Implicit Imitation Guidancearxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [2206.11795] Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videosarxiv.org
• Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Trainingalphaxiv.org
• [2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipearxiv.org