Eric Zhang on X: "So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - https://t.co/0HNE3opwWm" / X
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B -
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - GPT-OSS - Inkling It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark Not sure if I’ll continue with this, but it was fun! The…
saved by
related reading
- Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
- Shipping at Inference-Speed | Peter Steinbergersteipete.me
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Spending Inference Time - Kevin Lukevinlu.ai
- ali (@waterloo_intern) on Xx.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- How GPT, Claude, and Gemini are actually trained and served – Reiner Popeyoutube.com
- >10x More Efficient Pretraining — Magicmagic.dev
- GitHub - JustVugg/colibri: Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦github.com
- Two different tricks for fast LLM inferenceseangoedecke.com