flâneur

[2305.14251] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

arxiv.org · 761 words · saved by 1 readers

Abstract:Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly. In this paper, we introduce FActScore (Factual precision in Atomicity Score), a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source. We conduct an extensive human evaluation to obtain FActScores of people biographies generated by several state-of-the-art commercial LMs -- InstructGPT, ChatGPT, and the retrieval-augmented PerplexityAI -- and report new analysis demonstrating the need for such a fine-grained score (e.g., ChatGPT only achieves 58%). Since human evaluation is costly, we also introduce an automated model that estimates FActScore, using retrieval and a strong language model, with less than a 2% error rate. Finally, we use this automated metric to evaluate 6,500 generations from a new set of 13 recent LMs that would have cost $26K if evaluated by humans, with various findings: GPT-4 and ChatGPT are more factual than public models, and Vicuna and Alpaca are some of the best public models.

[2305.14251] FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation --> Computer Science > Computation and Language arXiv:2305.14251 (cs) [Submitted on 23 May 2023 ( v1 ), last revised 11 Oct 2023 (this version, v2)] Title: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation Authors: Sewon Min , Kalpesh Krishna , Xinxi Lyu , Mike Lewis , Wen-tau Yih , Pang Wei Koh , Mohit Iyyer , Luke Zettlemoyer , Hannaneh Hajishirzi View a PDF of the paper titled FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long

saved by

related reading