flâneur

Extending the Context of Pretrained LLMs by Dropping their Positional Embeddings

pub.sakana.ai · 3,257 words · saved by 1 readers

Many valuable real-world tasks are long: reviewing a change that touches many files, continuing a month-old chat, or answering questions about a 200-page contract or an hours-long transcript. In these settings, the useful details often lie far into the provided context, and the model must keep track of names, variables, assumptions, instructions, or feedback across long stretches of text. Today’s strongest models do offer large context windows, but their accuracy and recall capabilities considerably suffer once we go past typical sequence lengths seen in training. “So, why don’t we just train on longer sequences?” Unfortunately, training language models on long sequences is not easy. Part of the difficulty lies in the data: truly long, clean, and relevant contexts are rare and expensive to curate. The other part is compute: attention compares every token to every other token, which means training costs grow quadratically with sequence length — making long-context training brutally expe

--> This page requires Javascript. Please enable it to view the website. --> RoPE --> DroPE Extending the Context of Pretrained LLMs by Dropping their Positional Embeddings tl;dr We introduce DroPE , a method for extending context windows without long-context fine-tuning. By removing positional embeddings and running a short recalibration, we achieve seamless context extension that maintains base performance while far outperforming RoPE scaling methods. Proven effective across scales up to 7B parameters and trillion-token datasets. --> Yoav Gelberg Sakana AI, University of Oxford Koshi Eguchi

saved by

related reading