Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)

Today, Kog AI launches a tech preview of the Kog Inference Engine (KIE): 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). This preview runs a 2B model, with support for large third-party MoE models coming next at similar speeds.

TL;DR: we show that AI inference on GPUs can be super-fast, reaching the speed regime of dedicated inference hardware cards when optimizing the whole software stack with architecture/engine/kernel co-design. Test the speed in our live coding playground: playground.kog.ai . This post explains why optimizing for single-request LLM decoding speed is important for AI agents; why it's primarily a memory-bandwidth maximization problem, not a FLOPS one; why standard datacenter GPU hardware has a much higher decoding-speed ceiling than current inference stacks expose due to software bottlenecks ; and

saved by

• Alex K

related reading

• Kimi-K2.5 Inference Benchmark - Luminalluminal.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• How To Scale Your Modeljax-ml.github.io
• How is LLaMa.cpp possible?finbarr.ca
• The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
• Optimizing inference · Hugging Facehuggingface.co
• A Guide to AI Inference Engineering - ByteByteGo Newsletterblog.bytebytego.com