RETRO Is Blazingly Fast | Mitchell A. Gordon
When I first read Google’s RETRO paper, I was skeptical. Sure, RETRO models are 25x smaller than the competition, supposedly leading to HUGE savings in training and inference costs. But what about the new trillion token “retrieval database” they added to the architcture? Surely that must add back some computational costs, balancing the cosmic seesaw?
When I first read Google’s RETRO paper, I was skeptical. Sure, RETRO models are 25x smaller than the competition, supposedly leading to HUGE savings in training and inference costs. But what about the new trillion token “retrieval database” they added to the architcture? Surely that must add back some computational costs, balancing the cosmic seesaw? Apparently not. After running benchmarks for myself, at scale, I am convinced that RETRO is indeed BLAZINGLY fast. RETRO is so fast and cheap, in fact, that I cannot fathom why anyone would choose to do language modeling without retrieval. RETRO O
Explore this link on the map →related reading
- Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
- Faiss: A library for efficient similarity search - Engineering at Metaengineering.fb.com
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- Improving language models by retrieving from trillions of tokensarxiv.org
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io
- AI Model & API Providers Analysis | Artificial Analysisartificialanalysis.ai
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
- Rerankers and Two-Stage Retrieval | Pineconepinecone.io
- Aman's AI Journal • Primers • Retrieval Augmented Generationaman.ai
- Inference characteristics of Llama · Cursorcursor.com