ali on X: "22580: From GPT2 to Kimi3, Explained" / X

https://t.co/aivc7B4A7Z

Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside KimiK3 (2026). We scaled up by a factor of 22,580 in seven years. But is it just... scale? In this worklog, I’ll walk through how we got here and how much, or how little, has actually changed since then. We’ll trace the major architectural developments leading to KimiK3. GPT-2 GPT-2 is a decoder-only architecture: The input receives token and positional embeddings: Each transformer block, zoomed in, looks like this: The attention process: Once the final hidden-state matrix is produced, the…

saved by

• Devin Plumb
• Roy Liao
• Ishaan Panigrahi
• SUJASH AGRAWAL

related reading

• How trillion-parameter AI models work and how to serve themyoutube.com
• How Kimi Trains 896 Experts Without Collapseyoutube.com
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Kimi K3 Tech Blog: Open Frontier Intelligencekimi.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org
• How LLMs Actually Work | 0xkato0xkato.xyz
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Kimi Linear: An Expressive, Efficient Attention Architecturealphaxiv.org
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• The Annotated Transformernlp.seas.harvard.edu