Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)
Today, Kog AI launches a tech preview of the Kog Inference Engine (KIE): 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). This preview runs a 2B model, with support for large third-party MoE models coming next at similar speeds.
TL;DR: we show that AI inference on GPUs can be super-fast, reaching the speed regime of dedicated inference hardware cards when optimizing the whole software stack with architecture/engine/kernel co-design. Test the speed in our live coding playground: playground.kog.ai . This post explains why optimizing for single-request LLM decoding speed is important for AI agents; why it's primarily a memory-bandwidth maximization problem, not a FLOPS one; why standard datacenter GPU hardware has a much higher decoding-speed ceiling than current inference stacks expose due to software bottlenecks ; and
saved by
related reading
- Kimi-K2.5 Inference Benchmark - Luminalluminal.com
- Compiling Models to Megakernels - by Luminal and Joe Fiotiblog.luminal.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Together AI | The AI Native Cloudtogether.ai
- Parallel CPU-GPU Execution for LLM Inference on Constrained GPUsarxiv.org
- A Guide to AI Inference Engineering - ByteByteGo Newsletterblog.bytebytego.com
- Groq Inference Tokenomics: Speed, But At What Cost?semianalysis.com
- Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRTbaseten.co
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io