LlamaCPP - LlamaIndex 🦙 0.8.49

Note that if you’re using a version of llama-cpp-python after version 0.1.79, the model format has changed from ggmlv3 to gguf. Old model files like the used in this notebook can be converted using scripts in the llama.cpp repo. Alternatively, you can download the GGUF version of the model above from huggingface. By default, if model_path and model_url are blank, the LlamaCPP module will load llama2-chat-13B in either format depending on your version. To get the best performance out of LlamaCPP, it is recomended to install the package so that it is compilied with GPU support. A full guide for installing this way is here. Full MACOS instructions are also here. In general: Use CuBLAS if you have CUDA and an NVidia GPU Use METAL if you are running on an M1/M2 MacBook Use CLBLAST if you are running on an AMD/Intel GPU The LlamaCPP llm is highly configurable. Depending on the model being used, you’ll want to pass in messages_to_prompt and completion_to_prompt functions to help format the mo

Note that if you’re using a version of llama-cpp-python after version 0.1.79, the model format has changed from ggmlv3 to gguf. Old model files like the used in this notebook can be converted using scripts in the llama.cpp repo. Alternatively, you can download the GGUF version of the model above from huggingface. By default, if model_path and model_url are blank, the LlamaCPP module will load llama2-chat-13B in either format depending on your version. To get the best performance out of LlamaCPP, it is recomended to install the package so that it is compilied with GPU support. A full guide for