RETRO Is Blazingly Fast | Mitchell A. Gordon

When I first read Google’s RETRO paper, I was skeptical. Sure, RETRO models are 25x smaller than the competition, supposedly leading to HUGE savings in training and inference costs. But what about the new trillion token “retrieval database” they added to the architcture? Surely that must add back some computational costs, balancing the cosmic seesaw?

When I first read Google’s RETRO paper, I was skeptical. Sure, RETRO models are 25x smaller than the competition, supposedly leading to HUGE savings in training and inference costs. But what about the new trillion token “retrieval database” they added to the architcture? Surely that must add back some computational costs, balancing the cosmic seesaw? Apparently not. After running benchmarks for myself, at scale, I am convinced that RETRO is indeed BLAZINGLY fast. RETRO is so fast and cheap, in fact, that I cannot fathom why anyone would choose to do language modeling without retrieval. RETRO O

related reading

• Improving language models by retrieving from trillions of tokensarxiv.org
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Spending Inference Time - Kevin Lukevinlu.ai
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• 2112.04426arxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Mosaic LLMs: GPT-3 quality formosaicml.com
• Mamba: The Easy Wayjackcook.com