The Secret Sauce behind 100K context window in LLMs: all tricks in one place | by Galina Alperovich | May, 2023 | GoPenAI

tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and…

Large Language Models Artificial Intelligence Machine Learning The Secret Sauce behind 100K context window in LLMs: all tricks in one place Galina Alperovich 16 min read · May 15, 2023 -- 5 Listen Share tldr; techniques to speed up training and inference of LLMs to use large context window up to 100K input tokens during training and inference: ALiBi positional embedding, Sparse Attention, FlashAttention, Multi-Query attention, Conditional computation, and 80GB A100 GPUs. Recently there were several announcements about new Large Language Models (LLMs) that can consume an extremely large context

related reading

• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• A History of Large Language Modelsgregorygundersen.com