✳flâneur — a map of the web's best reading

Late Chunking in Long-Context Embedding Models

jina.ai · 1,754 words · saved by 1 readers

About a year ago, in October 2023, we released the world's first open-source embedding model with an 8K context length, jina-embeddings-v2-base-en. Since then, there has been quite some debate about the usefulness of long-context in embedding models. For many applications, encoding a document thousands of words long into a single embedding representation is not ideal. Many use cases require retrieving smaller portions of the text, and dense vector-based retrieval systems often perform better with smaller text segments, as the semantics are less likely to be "over-compressed" in the embedding vectors. Retrieval-Augmented Generation (RAG) is one of the most well-known applications that requires splitting documents into smaller text chunks (say within 512 tokens). These chunks are usually stored in a vector database, with vector representations generated by a text embedding model. During runtime, the same embedding model encodes a query into a vector representation, which is then used to

💡 Late Chunking is now available in jina-embeddings-v3 API. Recommended reading order: part I, part II , research paper . What Late Chunking Really Is & What It’s Not: Part II Part 2 of our exploration of Late Chunking, a deep dive into why it is the best method for chunk embeddings and improving search/RAG performance. New! Part II: deep dive into boundary cues and misconception. About a year ago, in October 2023, we released the world's first open-source embedding model with an 8K context length , jina-embeddings-v2-base-en . Since then, there has been quite some debate about the usefulness

Explore this link on the map →

related reading