The Secret Sauce behind 100K context window in LLMs: all tricks in one place | by Galina Alperovich | May, 2023 | GoPenAI

tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and…

Large Language Models Artificial Intelligence Machine Learning The Secret Sauce behind 100K context window in LLMs: all tricks in one place Galina Alperovich 16 min read · May 15, 2023 -- 5 Listen Share tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and inference: ALiBi positional embedding, Sparse Attention, FlashAttention, Multi-Query attention, Conditional computation, and 80GB A100 GPUs. Recently there were several announcements about new Large Language Models (LLMs) that can consume an extremely large context

related reading

• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• 2502.11089arxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com