flâneur

LLM Chess: Benchmarking Reasoning and Instruction-Following in LLMs through Chess

arxiv.org · 10,918 words · saved by 1 readers

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on. Authors: achieve the best HTML results from your LaTeX submissions by following these best practices. We introduce LLM Chess, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, m

LLM Chess : Benchmarking Reasoning and Instruction-Following in LLMs through Chess Sai Kolasani 1 , Maxim Saplin 2 , Nicholas Crispino 3 , Kyle Montgomery 3 , Jared Quincy Davis 4 , Matei Zaharia 1 , Chi Wang 5 , Chenguang Wang 3 1 UC Berkeley, 2 Independent Researcher, 3 UC Santa Cruz, 4 Stanford University, 5 Google DeepMind saikolasani@berkeley.edu , tutehabre@gmail.com Abstract We introduce LLM Chess , an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the doma

saved by

related reading