How Big a Deal are MatMul-Free Transformers? — LessWrong

If you’re already familiar with the technical side of LLMs, you can skip the first section. …

x How Big a Deal are MatMul-Free Transformers? — LessWrong Academic Papers Transformers AI Frontpage 19 How Big a Deal are MatMul-Free Transformers? by JustisMills 27th Jun 2024 Linkpost for justismills.substack.com 6 min read 6 19 If you’re already familiar with the technical side of LLMs, you can skip the first section. The story so far Modern Large Language Models - your ChatGPTs, your Geminis - are a particular kind of transformer , a deep learning architecture invented about seven years ago . Without getting into the weeds , transformers basically work by turning an input into numbers, an

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• How To Scale Your Modeljax-ml.github.io
• Scalable MatMul-free Language Modelingarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Can LLMs Be Computers?percepta.ai
• Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com
• AI Revolution - Transformers and Large Language Models (LLMs)blog.eladgil.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• A History of Large Language Modelsgregorygundersen.com
• Bits, FLOPS, and Watts: A Systems-Level Perspective of Scaling LLMs — Part 1 | by Asheesh Goja | Mediummedium.com