Can Longer Sequences Help Take the Next Leap in AI? | SAIL Blog

Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements.

Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements. There is a good reason: the ubiquitous Transformer model is an absolute wonder, but it is difficult to scale in sequence length for both performance and quality reasons. For complexity nerds, the Transformer’s runtime is quadratic

related reading

• From Deep to Long Learning? · Hazy Researchhazyresearch.stanford.edu
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• [2011.04006] Long Range Arena: A Benchmark for Efficient Transformersarxiv.org
• [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• GLM-5.2: Built for Long-Horizon Tasksz.ai