[2311.09682] MacGyver: Are Large Language Models Creative Problem Solvers?

Abstract:We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking. This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI.

View PDF HTML (experimental) Abstract:We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar…

saved by

• Florent Tavernier

related reading

• Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputarxiv.org
• Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org
• the-illusion-of-thinking.pdfml-site.cdn-apple.com
• Language Models can Solve Computer Tasksarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Can LLMs generate novel research ideas?arxiv.org
• Art or Artifice? Large Language Models and the False Promise of Creativityarxiv.org
• Program Synthesis with Large Language Modelsarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• Scaling Self-Play with Self-Guidancearxiv.org
• Creativity Has Left the Chat: The Price of Debiasing Language Modelsarxiv.org
• LMCA_dataset.pdfandrew.cmu.edu