[2605.13408] From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks
Abstract:In this paper, we examine linguistic puzzles used in high school linguistics competitions, focusing on two common formats: Rosetta Stone and Match-Up. We propose a systematic procedure for converting existing Rosetta Stone puzzles into corresponding Match-Up counterparts. Because linguistic puzzle creation is complex and time-consuming, our method provides an efficient way to accelerate the generation of new puzzles. We evaluate the resulting Rosetta Stone-Match-Up pairs with both human participants and large language models (LLMs). Our results show that both expert human solvers and LLMs display an all-or-nothing pattern on Match-Up puzzles, either solving them completely or failing entirely. This work contributes a new dataset of paired puzzles and provides a detailed evaluation of puzzle difficulty across formats, offering insights into both human and machine linguistic reasoning.
View PDF HTML (experimental) Abstract:In this paper, we examine linguistic puzzles used in high school linguistics competitions, focusing on two common formats: Rosetta Stone and Match-Up. We propose a systematic procedure for converting existing Rosetta Stone puzzles into corresponding Match-Up counterparts. Because linguistic puzzle creation is complex and time-consuming, our method provides an efficient way to accelerate the generation of new puzzles. We evaluate the resulting Rosetta Stone-Match-Up pairs with both human participants and large language models (LLMs). Our results show…
saved by
related reading
- 2026.mellm-1.2.pdfaclanthology.org
- Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscationarxiv.org
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io
- the-illusion-of-thinking.pdfml-site.cdn-apple.com
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
- [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
- MacGyver: Are Large Language Models Creative Problem Solvers?arxiv.org
- Language Model Compositionarxiv.org
- 2301.06627arxiv.org