LongLLMLingua: Bye-bye to Middle Loss and Save on Your RAG Costs via Prompt Compression | by Huiqiang Jiang | Nov, 2023 | LlamaIndex Blog

In the RAG, after the retrieval phase, it’s necessary to perform Re-ranking + Fine-Grained Prompt Compression + Subsequence Recovery to enhance LLM’s perception of key information, which is LongLLMLingua. TL;DR: While Retrieval-Augmented Generation (RAG) is highly effective in various scenarios, it still has drawbacks such as 1) Performance drop, like the “Lost in the middle” issue, 2) High costs, both financially and in terms of latency, and 3) Context windows limitation. LongLLMLingua offers a solution to these problems in RAG or Long Context scenarios via prompt compression. It can boost accuracy by as much as 21.4% while only using ¼ of the tokens. In long context situations, it can save $28 for every 1000 examples. See real-world cases on the project page. We previously wrote a blog post introducing the design of LLMLingua, which started from the perspective of designing a special language for LLMs. This time, our focus will be on the scenarios involving RAG. Retrieval-Augmented G

Follow us on In the RAG, after the retrieval phase, it’s necessary to perform Re-ranking + Fine-Grained Prompt Compression + Subsequence Recovery to enhance LLM’s perception of key information, which is LongLLMLingua. Ready to get started with LlamaParse? Explore our free and paid plans today. Learn more TL;DR: While Retrieval-Augmented Generation (RAG) is highly effective in various scenarios, it still has drawbacks such as 1) Performance drop, like the “Lost in the middle” issue, 2) High costs, both financially and in terms of latency, and 3) Context windows limitation. LongLLMLingua offers

saved by

• Eric Chang

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
• Retrieval-Augmented Generation for Large Language Models: A Surveyarxiv.org
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Advanced Retriever Techniques to Improve Your RAGs | Towards Data Sciencetowardsdatascience.com
• Long Context RAG Performance of LLMs | Databricks Blogdatabricks.com
• Advanced RAG Techniques: What They Are & How to Use Themfalkordb.com
• Rerankers and Two-Stage Retrieval | Pineconepinecone.io
• What is Retrieval Augmented Generation (RAG)? | Databricksdatabricks.com
• LLMs can invent their own compression - Rajan Agarwalrajan.sh
• State of RL for reasoning LLMs | A. Weersaweers.de
• Vinija's Notes • NLP • Retrieval Augmented Generationvinija.ai