Eric Zhang on X: "So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - https://t.co/0HNE3opwWm" / X
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B -
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - GPT-OSS - Inkling It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark Not sure if I’ll continue with this, but it was fun! The…
saved by
related reading
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Composer2.pdfcursor.com
- Together AI | The AI Native Cloudtogether.ai
- As Rocks May Think | Eric Jangevjang.com
- Spending Inference Time - Kevin Lukevinlu.ai
- GitHub - wafer-ai/gpu-perf-engineering-resources: A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.github.com
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Inferactinferact.ai
- Kevin-32B: Multi-Turn RL for Writing CUDA Kernels | Cognitioncognition.ai
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- Optimizing inference · Hugging Facehuggingface.co