[2411.00062] Evolving Alignment via Asymmetric Self-Play

arxiv.org · 5,072 words · saved by 1 readers

Abstract:Current RLHF frameworks for aligning large language models (LLMs) typically assume a fixed prompt distribution, which is sub-optimal and limits the scalability of alignment and generalizability of models. To address this, we introduce a general open-ended RLHF framework that casts alignment as an asymmetric game between two players: (i) a creator that generates increasingly informative prompt distributions using reward signals, and (ii) a solver that learns to produce more preferred responses on prompts produced by the creator. This framework of Evolving Alignment via Asymmetric Self-Play (eva), results in a simple and efficient approach that can utilize any existing RLHF algorithm for scalable alignment. eva outperforms state-of-the-art methods on widely-used benchmarks, without the need of any additional human crafted prompts. Specifically, eva improves the win rate of Gemma-2-9B-it on Arena-Hard from 51.6% to 60.1% with DPO, from 55.7% to 58.9% with SPPO, from 52.3% to 60.7% with SimPO, and from 54.8% to 60.3% with ORPO, surpassing its 27B version and matching claude-3-opus. This improvement is persistent even when new human crafted prompts are introduced. Finally, we show eva is effective and robust under various ablation settings.

Scalable Reinforcement Post-Training Beyond Static Human Prompts Evolving Alignment via Asymmetric Self-Play Ziyu Ye12 Rishabh Agarwal1 Tianqi Liu1 Rishabh Joshi1 Sarmishta Velury1 Quoc V. Le1 Qijun Tan1 Yuan Liu1 Abstract The Evolving Alignment Principle Existing reinforcement post-training pipeline for arXiv:2411.00062v3 [cs.CL] 9 Apr 2025…

[2411.00062] Evolving Alignment via Asymmetric Self-Play

related reading