The Secret Sauce behind 100K context window in LLMs: all tricks in one place | by Galina Alperovich | May, 2023 | GoPenAI

tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and…

Large Language Models Artificial Intelligence Machine Learning The Secret Sauce behind 100K context window in LLMs: all tricks in one place Galina Alperovich 16 min read · May 15, 2023 -- 5 Listen Share tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and inference: ALiBi positional embedding, Sparse Attention, FlashAttention, Multi-Query attention, Conditional computation, and 80GB A100 GPUs. Recently there were several announcements about new Large Language Models (LLMs) that can consume an extremely large context

related reading

• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• Subquadratic — How SSA Makes Long Context Practicalsubq.ai
• 2502.11089arxiv.org
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• What is a context window? | IBMibm.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev