flâneur

[2310.08901] Welfare Diplomacy: Benchmarking Language Model Cooperation

arxiv.org · 639 words · saved by 1 readers

Abstract:The growing capabilities and increasingly widespread deployment of AI systems necessitate robust benchmarks for measuring their cooperative capabilities. Unfortunately, most multi-agent benchmarks are either zero-sum or purely cooperative, providing limited opportunities for such measurements. We introduce a general-sum variant of the zero-sum board game Diplomacy -- called Welfare Diplomacy -- in which players must balance investing in military conquest and domestic welfare. We argue that Welfare Diplomacy facilitates both a clearer assessment of and stronger training incentives for cooperative capabilities. Our contributions are: (1) proposing the Welfare Diplomacy rules and implementing them via an open-source Diplomacy engine; (2) constructing baseline agents using zero-shot prompted language models; and (3) conducting experiments where we find that baselines using state-of-the-art models attain high social welfare but are exploitable. Our work aims to promote societal safety by aiding researchers in developing and assessing multi-agent AI systems. Code to evaluate Welfare Diplomacy and reproduce our experiments is available at this https URL.

[2310.08901] Welfare Diplomacy: Benchmarking Language Model Cooperation Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Multiagent Systems arXiv:2310.08901 (cs) [Submitted on 13 Oct 2023] Title: Welfare Diplomacy: Benchmarking Language Model Cooperation Authors: Gabriel Mukobi , Hannah Erlebach , Niklas Lauffer , Lewis Hammond , Alan Chan , Jesse Clifton View a PDF of the paper titled Welfare Diplomacy: Benchmarking Language Model Cooperation, by Gabriel Mukobi and 5 other authors

related reading