Extending Context is Hard | kaiokendev.github.io

pages

Extending Context is Hard | kaiokendev kaiokendev Extending Context is Hard…but not Impossible† On the surface, it should be an easy task. I was working on this write up while I was working through methods to finetune the pre-trained model for longer sequence length. In this case, the pre-trained model is LLaMa, with a pre-training sequence length of 2048. Naively fine-tuning the model on long sequences never seemed to work, but I felt it must be possible, so I stubbornly pushed through it. Now, there is a way to extend context with just 1 line of code, and it is getting a lot of attention. Un

saved by

• Atem Aguer
• Jirat C
• Amos You

related reading

• [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
• MaroonLLM Maybe MaroonLongLM: SelfExtend LLM Context Window Without Tuningarxiv.org
• Mediumblog.gopenai.com
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• A History of Large Language Modelsgregorygundersen.com