[2201.02387] The Defeat of the Winograd Schema Challenge

The Winograd Schema Challenge -- a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge -- was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and assess its significance.

The Defeat of the Winograd Schema Challenge Vid Kocijana,1 , Ernest Davisb , Thomas Lukasiewiczc,d , Gary Marcuse , Leora Morgensternf a Kumo.ai, 357 Castro Street, Suite 200 Mountain View, CA 94041, United States b New York University, Department of Computer Science, 251 Mercer…

related reading

• Winograd schema challenge - Wikipediaen.wikipedia.org
• What Does It Mean for AI to Understand? | Quanta Magazinequantamagazine.org
• As Rocks May Think | Eric Jangevjang.com
• Language Models can Solve Computer Tasksarxiv.org
• the-illusion-of-thinking.pdfml-site.cdn-apple.com
• Group | Sherry Tongshuang Wucs.cmu.edu
• [2406.02061] Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelsarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Trending Papers - Hugging Facepaperswithcode.com
• 2312.00688arxiv.org
• BIG-bench/bigbench/benchmark_tasks/keywords_to_tasks.md at main · google/BIG-benchgithub.com
• Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Searchprimeintellect.ai