Jamba: A Hybrid Transformer-Mamba Language Model

Finally, the first production-grade commercially available Mamba-based model delivering best-in-class quality and performance is here. Introducing Jamba, a novel architecture which combines Attention and Mamba layers, with MoE modules. Here are some key features of Jamba So how does Jamba provide flexibility for balancing performance and memory requirements, while the previous Mamba-Transformer based models were not able to do so ? It all comes down to Jamba architecture choice, which aims to provide not only a small number of active parameters but also an 8x smaller KV cache compared to a vanilla Transformer. The basic component is a Jamba block, which may be repeated in sequence. Each Jamba block is a combination of Mamba or Attention layers. Each such layer contains either an attention or a Mamba module, followed by a multi-layer perceptron (MLP). Further, In Jamba some of the MLPs may be replaced by MoE layers, which helps increase the model capacity while keeping the active number

Santosh Sawant - Jamba: A Hybrid Transformer-Mamba Language Model Finally, the first production-grade commercially available Mamba-based model delivering best-in-class quality and performance is here. Introducing Jamba, a novel architecture which combines Attention and Mamba layers, with MoE modules. Here are some key features of Jamba 3X throughput on long contexts compared to Mixtral 8x7B Democratizes access to a massive 256K context window The only model in its size class that fits up to 140K context on a single GPU So how does Jamba provide flexibility for balancing performance and memory

related reading

• Mamba: The Easy Wayjackcook.com
• Mamba Explainedthegradient.pub
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Mamba No. 5 (A Little Bit Of…) | Sparse Notesjameschen.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• MambaByte: Token-free Selective State Space Modelarxiv.org
• A Visual Guide to Mamba and State Space Modelsnewsletter.maartengrootendorst.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Mamba Effect: State Space Models Taking on Transformershungleai.substack.com