How is LLaMa.cpp possible? - by Finbarr Timbers

An exercise in applied inference arithmetic

Note: Substack doesn’t have great support for LaTeX, so you might want to read this article on my blog instead. Recently, a project rewrote the LLaMa inference code in raw C++. With some optimizations and by quantizing the weights, the project allows running LLaMa locally on a wild variety of hardware: On a Pixel5, you can run the 7B parameter model at 1 tokens/s. On a M2 Macbook Pro, you can get ~16 tokens/s with the 7B parameter model You can even run the 7B model on a 4GB RAM Raspberry Pi, albeit at 0.1 tokens/s. If you are like me, you saw this and thought: What? How is this…

related reading

• How is LLaMa.cpp possible?finbarr.ca
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• A guide to LLM inference and performancebaseten.co
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How To Scale Your Modeljax-ml.github.io
• Making Deep Learning go Brrrr From First Principleshorace.io
• LLM Inference Economics from First Principlestensoreconomics.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io