[2309.14556] Art or Artifice? Large Language Models and the False Promise of Creativity
Abstract:Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT), which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose the Torrance Test of Creative Writing (TTCW) to evaluate creativity as a product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.
View PDF HTML (experimental) Abstract:Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT), which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose the Torrance Test of Creative Writing (TTCW) to evaluate creativity as a product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality,…
saved by
related reading
- Creativity Has Left the Chat: The Price of Debiasing Language Modelsarxiv.org
- Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputarxiv.org
- Can LLMs generate novel research ideas?arxiv.org
- [2603.00452] Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writingarxiv.org
- Modifying Large Language Model Post-Training for Diverse Creative Writingarxiv.org
- [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
- MacGyver: Are Large Language Models Creative Problem Solvers?arxiv.org
- Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
- Rotating the Space: On LLMs as a Medium for Thought | rotating_the_spacesbgeoaiphd.github.io
- 2301.06627arxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- StoryScope: Investigating idiosyncrasies in AI fictionarxiv.org