Automated Metadata Extraction for Better Retrieval + Synthesis - LlamaIndex 🦙 0.8.29.post1
In this tutorial, we show you how to perform automated metadata extraction for better retrieval results. We use two extractors: a QuestionAnsweredExtractor which generates question/answer pairs from a piece of text, and also a SummaryExtractor which extracts summaries, not only within the current text, but also within adjacent texts. We show that this allows for “chunk dreaming” - each individual chunk can have more “holistic” details, leading to higher answer quality given retrieved results. Our data source is taken from Eugene Yan’s popular article on LLM Patterns: https://eugeneyan.com/writing/llm-patterns/ Here we define metadata extractors. We define two variants: metadata_extractor_1 only contains the QuestionsAnsweredExtractor metadata_extractor_2 contains both the QuestionsAnsweredExtractor as well as the SummaryExtractor We also show how to instantiate the SummaryExtractor and QuestionsAnsweredExtractor. We load in Eugene’s essay (https://eugeneyan.com/writing/llm-patterns/) u
Explore this link on the map →