[2412.00543] Evaluating the Consistency of LLM Evaluators
Abstract:Large language models (LLMs) have shown potential as general evaluators along with the evident benefits of speed and cost. While their correlation against human annotators has been widely studied, consistency as evaluators is still understudied, raising concerns about the reliability of LLM evaluators. In this paper, we conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency (IC), on different scoring scales and criterion granularity with open-source and proprietary models. Our comprehensive analysis demonstrates that strong proprietary models are not necessarily consistent evaluators, highlighting the importance of considering consistency in assessing the capability of LLM evaluators.
Abstract:Large language models (LLMs) have shown potential as general evaluators along with the evident benefits of speed and cost. While their correlation against human annotators has been widely studied, consistency as evaluators is still understudied, raising concerns about the reliability of LLM evaluators. In this paper, we conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency (IC), on different scoring scales and criterion granularity with open-source and proprietary models. Our comprehensive analysis demonstrate
Explore this link on the map →saved by
related reading
- The bitter lesson of LLM evalsparsed.com
- Applying Statistics to LLM Evaluationscameronrwolfe.substack.com
- Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
- LLM Evaluation doesn't need to be complicatedphilschmid.de
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- LLM-as-a-judge: a complete guide to using LLMs for evaluationsevidentlyai.com
- Position: It's Time to Optimize for Self-Consistencytime-for-consistency.github.io
- Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
- Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
- LLM evaluation: a beginner's guideevidentlyai.com
- [2503.07003] Large Language Models Often Say One Thing and Do Anotherarxiv.org
- LLM-as-a-Verifier: A General-Purpose Verification Framework | alphaXivalphaxiv.org