Big Boss on X: "First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things. As an example," / X

First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things. As an example,

First you have to understand that modern LLM inference already disaggregates weights as models outgrew single chips years ago. You shard either by layer (pipeline parallelism) or by slicing every layer (tensor parallelism), and the two do very different things. As an example, let’s look at Llama 3.3. It has 70B of weights and at FP8 that’s 70 GB of memory which is enough to fit on a single H100. Now that H100 has 3.35 TB/s of HBM, so the fastest it can ever decode for one user is 70/3.35 ≈ 21 ms/token or ~48 tok/s while using under 1% of its FLOPs. Now if we pipeline it across 8 chips: each…

saved by

• Ishaan Panigrahi

related reading

• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How To Scale Your Modeljax-ml.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How is LLaMa.cpp possible?finbarr.ca
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• LLM Inference Economics from First Principlestensoreconomics.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev