flâneur

Long-Context Retrieval Models with Monarch Mixer · Hazy Research

hazyresearch.stanford.edu · 3,609 words · saved by 1 readers

Text embeddings are a critical piece of many pipelines, from search, to RAG, to vector databases and more. Most embedding models are BERT/Transformer-based and typically have short context lengths (e.g., 512). That’s only about two pages of text, but documents can be very long – books, legal cases, TV screenplays, code repositories, etc can be tens of thousands of tokens long (or more). Here, we’re taking a first step towards developing long-context retrieval models. We build on Monarch Mixer (M2), a recent model family developing attention- and MLP-free BERT models, which are enabling long-context BERT models. Today, we’re releasing a preview of a few models: long-context versions of M2-BERT up to 32K context length, as well as embedding versions fine-tuned for long-context retrieval. In this blog, you’ll hear about the changes to data mixture and loss function we needed to enable these new models. We’re also releasing the V0 of a long-context retrieval benchmark called LoCo, that we’

Long-Context Retrieval Models with Monarch Mixer · Hazy Research Jan 11, 2024 · 17 min read Long-Context Retrieval Models with Monarch Mixer Jon Saad-Falcon , Dan Fu , Simran Arora Text embeddings are a critical piece of many pipelines, from search, to RAG, to vector databases and more. Most embedding models are BERT/Transformer-based and typically have short context lengths (e.g., 512). That’s only about two pages of text, but documents can be very long – books, legal cases, TV screenplays, code repositories, etc can be tens of thousands of tokens long (or more). Here, we’re taking a first st

related reading