flâneur

FutureHouse

futurehouse.org · 3,918 words · saved by 1 readers

We’ve been on a journey to build the best RAG system as judged by accuracy. Forget cost. Forget latency. Here we show experiments that led to the design of PaperQA2, FutureHouse’s scientific RAG system, which exceeds scientist’s performance on tasks like answering challenging scientific questions, writing review articles, and detecting contradictions from the literature. PaperQA2's high-accuracy design goal results in a different implementation from other commercial RAG systems. What we found to be important for RAG accuracy:‍ What we found to be unimportant for RAG accuracy: We measured the impact of our PaperQA2 design decisions using metrics based on LitQA2, a set of 200 expert-crafted multiple-choice questions. Correct answers require comprehension of intermediate results within the full-text of recent scientific papers. The questions are designed to be specific enough that they can only be answered with a single source. Each question has an “Insufficient Information” option, which

We’ve been on a journey to build the best RAG system as judged by accuracy. Forget cost. Forget latency. Here we show experiments that led to the design of PaperQA2 , FutureHouse’s scientific RAG system, which exceeds scientists' performance on tasks like answering challenging scientific questions , writing review articles , and detecting contradictions from the literature . PaperQA2's high-accuracy design goal results in a different implementation from other commercial RAG systems. What we found to be important for RAG accuracy: ‍ An agentic approach, allowing for iterative query expansion. L

related reading