Extending the Context of Pretrained LLMs by Dropping their Positional Embeddings

Many valuable real-world tasks are long: reviewing a change that touches many files, continuing a month-old chat, or answering questions about a 200-page contract or an hours-long transcript. In these settings, the useful details often lie far into the provided context, and the model must keep track of names, variables, assumptions, instructions, or feedback across long stretches of text. Today’s strongest models do offer large context windows, but their accuracy and recall capabilities considerably suffer once we go past typical sequence lengths seen in training. “So, why don’t we just train on longer sequences?” Unfortunately, training language models on long sequences is not easy. Part of the difficulty lies in the data: truly long, clean, and relevant contexts are rare and expensive to curate. The other part is compute: attention compares every token to every other token, which means training costs grow quadratically with sequence length — making long-context training brutally expe

--> This page requires Javascript. Please enable it to view the website. --> RoPE --> DroPE Extending the Context of Pretrained LLMs by Dropping their Positional Embeddings tl;dr We introduce DroPE , a method for extending context windows without long-context fine-tuning. By removing positional embeddings and running a short recalibration, we achieve seamless context extension that maintains base performance while far outperforming RoPE scaling methods. Proven effective across scales up to 7B parameters and trillion-token datasets. --> Yoav Gelberg Sakana AI, University of Oxford Koshi Eguchi

saved by

• Rajan Agarwal

related reading

• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
• Arcee AI | Extending AFM-4.5B to 64k Context Lengtharcee.ai
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
• How LLMs Actually Work | 0xkato0xkato.xyz
• Mediumblog.gopenai.com