How is LLaMa.cpp possible?
If you want to read more of my writing, I have a Substack. Articles will be posted simultaneously to both places. Recently, a project rewrote the LLaMa inference code in raw C++. With some optimizations and quantizing the weights, this allows running a LLM locally on a wild variety of hardware: If you are like me, you saw this and thought: What? How is this possible? Don’t large models require expensive GPUs? I took my confusion and dove into the math surrounding inference requirements to understand the constraints we’re dealing with. Let’s start with GPUs. GPUs have two main benefits for deep learning: When we talk about memory bandwidth, we’re talking about how long it takes to move things from the HBM memory (i.e. the RAM) into the on-chip memory. To actually do math with the GPU, we need to move the matrices in question into the on-chip memory, which is quite small (40MB on an A100, compared to 40-80GB of RAM). Note that the memory bandwidth is ~2 orders of magnitude smaller than t
How is LLaMa.cpp possible? Finbarr Timbers Blog Books Advice LLM Performance Tools How is LLaMa.cpp possible? If you want to read more of my writing, I have a Substack . Articles will be posted simultaneously to both places. Note: This was written in March of '23, and is out of date (AI moves quickly!). This is an attempt at answering the question "How is it possible to run Llama on a single CPU?" and is not an attempt at documenting the current status of the Llama.cpp project. You should not rely on any of this post for specific details on how Llama.cpp functions. Recently, a project rewrote
Explore this link on the map →saved by
related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- A guide to LLM inference and performancebaseten.co
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- How To Scale Your Modeljax-ml.github.io
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- LLM Inference Economics from First Principlestensoreconomics.com
- Inference characteristics of Llama · Cursorcursor.com
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io