[2504.09389] Measuring LLM Novelty As The Frontier Of Original And High-Quality Output

Abstract:As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluates novelty as originality with respect to model training data, but original outputs may be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality -- the harmonic mean of the fraction of \ngrams unseen during training and a task-specific quality score. Using this framework, we identify trends that affect the novelty of generations from three families of open-data models (OLMo, OLMo-2, and Pythia) on three creative tasks: story completion, poetry writing, and creative tool use. We find that model-generated text from some base LLMs is less novel than human-written text from the internet. However, increasing model scale and post-training reliably improves novelty due to improvements in output quality. We also find that improving the base model at the same scale (\eg OLMo 7B to OLMo-2 7B) leads to higher novelty due to higher originality. Finally, we observe that inference-time methods, such as prompting and providing novel in-context examples, have a much smaller effect on novelty, often increasing originality at the expense of quality. This highlights the need for further research into more effective elicitation strategies as we use models for creative applications.

View PDF HTML (experimental) Abstract:As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluates novelty as originality with respect to model training data, but original outputs may be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality -- the harmonic mean of the…

saved by

• Florent Tavernier

related reading

• Modifying Large Language Model Post-Training for Diverse Creative Writingarxiv.org
• Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org
• Can LLMs generate novel research ideas?arxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Art or Artifice? Large Language Models and the False Promise of Creativityarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• Creativity Has Left the Chat: The Price of Debiasing Language Modelsarxiv.org
• [2409.19759] Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMsarxiv.org
• [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• [2508.19227] Generative Interfaces for Language Modelsarxiv.org