⭐️ Fast LLM Inference From Scratch

This post is about building an LLM inference engine using C++ and CUDA from scratch without libraries. Why? In doing so, we can learn about the full stack of LLM inference - which is becoming increasingly important1 - from CUDA kernels to model architecture, and get a real sense of how different optimizations affect inference speed. And one of the most important use cases is running fast on a single prompt on consumer devices. That’s what we’ll focus on: building a program that can load weights of common open models and do single-batch inference on them on a single CPU + GPU server, and iteratively improving the token throughput until it surpasses llama.cpp. Readers should have basic familiarity with large language models, attention, and transformers. The full source code is available on GitHub: yalm (Yet Another Language Model). Let’s recap how LLMs work, starting with their architecture and then moving onto inference mechanics. This will provide a starting point for an optimized impl

Contents Fast LLM Inference From Scratch Pushing single-GPU inference throughput to the edge without libraries Source code for this article on GitHub . Discussion on Hacker News . This post is about building an LLM inference engine using C++ and CUDA from scratch without libraries. Why? In doing so, we can learn about the full stack of LLM inference - which is becoming increasingly important Especially as inference compute becomes a new axis with which AI models scale, and models are increasingly deployed locally to devices on the edge. - from CUDA kernels to model architecture, and get a real

saved by

• Kunal Mishra
• Tasha Pais
• Dhruv Sheth
• Davis Brown

related reading

• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Acceleratorsarxiv.org
• How is LLaMa.cpp possible?finbarr.ca
• Optimizing inference · Hugging Facehuggingface.co
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• How To Scale Your Modeljax-ml.github.io
• How to make LLMs go fastvgel.me
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai