✳flâneur — a map of the web's best reading
LoRA Without Regret - Thinking Machines Lab
thinkingmachines.ai · 5,825 words · saved by 32 readers
How LoRA matches full training performance more broadly than expected.
Today's leading language models contain upwards of a trillion parameters, pretrained on tens of trillions of tokens. Base model performance keeps improving with scale, as these trillions are necessary for learning and representing all the patterns in written-down human knowledge. In contrast, post-training involves smaller datasets and generally focuses on narrower domains of knowledge and ranges of behavior. It seems wasteful to use a terabit of weights to represent updates from a gigabit or megabit of training data. This intuition has motivated parameter efficient fine-tuning (PEFT), which a
Explore this link on the map →saved by
- Winnie Xu
- Krish Shah
- Ratan Kaliani
- Freeman Jiang
- Emma Guo
- anka hu
- Hangyul Lyna Kim
- Uzay Girit
- Aaron Pham
- Dhruv Gautam
- Lydia Nottingham
- Sebastian J Zhao
related reading
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Anatomy of a Modern Finetuning APIbenanderson.work
- LLM Resourcesforrestbicker.com
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai