XetHub Blog | You Don't Need a Vector Database

Large Language models (LLMs) currently all have a bounded context size (~16K for GPT3.5), which poses a challenge if I want to use it to perform question answering on a very large collection of documents. We simply cannot inject all the documents into the context. One potential solution is to fine-tune the LLM on the set of documents before expecting useful responses to my prompts. However, finetuning LLM’s is difficult and costly if the set of documents is changing very quickly. For example, if I want to run Llama on my laptop so I can easily find emails, documents, etc, I certainly do not want to finetune Llama on every email received. The alternative solution is to use Retrieval Augmented Generation (RAG), which breaks down into two stages. First, we quickly search for the small subset of relevant documents in the total set of potential documents and then we include the contents of that document subset into the prompt before sending it to the LLM We can break down the first task int

Large Language models (LLMs) currently all have a bounded context size (~16K for GPT3.5), which poses a challenge if I want to use it to perform question answering on a very large collection of documents. We simply cannot inject all the documents into the context. One potential solution is to fine-tune the LLM on the set of documents before expecting useful responses to my prompts. However, finetuning LLM’s is difficult and costly if the set of documents is changing very quickly. For example, if I want to run Llama on my laptop so I can easily find emails, documents, etc, I certainly do not wa