Long-Context Retrieval Models with Monarch Mixer · Hazy Research

Text embeddings are a critical piece of many pipelines, from search, to RAG, to vector databases and more. Most embedding models are BERT/Transformer-based and typically have short context lengths (e.g., 512). That’s only about two pages of text, but documents can be very long – books, legal cases, TV screenplays, code repositories, etc can be tens of thousands of tokens long (or more). Here, we’re taking a first step towards developing long-context retrieval models. We build on Monarch Mixer (M2), a recent model family developing attention- and MLP-free BERT models, which are enabling long-context BERT models. Today, we’re releasing a preview of a few models: long-context versions of M2-BERT up to 32K context length, as well as embedding versions fine-tuned for long-context retrieval. In this blog, you’ll hear about the changes to data mixture and loss function we needed to enable these new models. We’re also releasing the V0 of a long-context retrieval benchmark called LoCo, that we’

Long-Context Retrieval Models with Monarch Mixer · Hazy Research Jan 11, 2024 · 17 min read Long-Context Retrieval Models with Monarch Mixer Jon Saad-Falcon , Dan Fu , Simran Arora Text embeddings are a critical piece of many pipelines, from search, to RAG, to vector databases and more. Most embedding models are BERT/Transformer-based and typically have short context lengths (e.g., 512). That’s only about two pages of text, but documents can be very long – books, legal cases, TV screenplays, code repositories, etc can be tens of thousands of tokens long (or more). Here, we’re taking a first st

related reading

• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Late Chunking in Long-Context Embedding Modelsjina.ai
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Arcee AI | Extending AFM-4.5B to 64k Context Lengtharcee.ai
• Generalizing an LLM from 8k to 1M Context using Qwen-Agent | Qwenqwenlm.github.io
• gemini_v1_5_report.pdfstorage.googleapis.com
• Long Context RAG Performance of LLMs | Databricks Blogdatabricks.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Subquadratic — How SSA Makes Long Context Practicalsubq.ai
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com