Can Longer Sequences Help Take the Next Leap in AI? | SAIL Blog
Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements.
Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements. There is a good reason: the ubiquitous Transformer model is an absolute wonder, but it is difficult to scale in sequence length for both performance and quality reasons. For complexity nerds, the Transformer’s runtime is quadratic
Explore this link on the map →related reading
- From Deep to Long Learning? · Hazy Researchhazyresearch.stanford.edu
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
- Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
- [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io
- GenAI Handbookgenai-handbook.github.io
- Arcee AI | Extending AFM-4.5B to 64k Context Lengtharcee.ai
- Subquadratic — How SSA Makes Long Context Practicalsubq.ai
- Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
- (Some of) The Models, They Just Don't Want to Learn | Tildeblog.tilderesearch.com