Can Longer Sequences Help Take the Next Leap in AI? | SAIL Blog
Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements.
Deep learning has revolutionized machine learning. To a first approximation, deeper has been better. However, there is another dimension to scale these models: the size of the input. Even the world’s most impressive models can only process long-form content by dismembering it into isolated, disconnected chunks of a few hundred words to fit their length requirements. There is a good reason: the ubiquitous Transformer model is an absolute wonder, but it is difficult to scale in sequence length for both performance and quality reasons. For complexity nerds, the Transformer’s runtime is quadratic
related reading
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- From Deep to Long Learning? · Hazy Researchhazyresearch.stanford.edu
- AI Revolution - Transformers and Large Language Models (LLMs)blog.eladgil.com
- [2011.04006] Long Range Arena: A Benchmark for Efficient Transformersarxiv.org
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- A History of Large Language Modelsgregorygundersen.com
- [2111.00396] Efficiently Modeling Long Sequences with Structured State Spacesarxiv.org
- Linear Transformers Are Faster After All – Manifest AImanifestai.com
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformersarxiv.org
- Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com