LoRA Without Regret - Thinking Machines Lab
thinkingmachines.ai · 5,825 words · saved by 37 readers
How LoRA matches full training performance more broadly than expected.
Today's leading language models contain upwards of a trillion parameters, pretrained on tens of trillions of tokens. Base model performance keeps improving with scale, as these trillions are necessary for learning and representing all the patterns in written-down human knowledge. In contrast, post-training involves smaller datasets and generally focuses on narrower domains of knowledge and ranges of behavior. It seems wasteful to use a terabit of weights to represent updates from a gigabit or megabit of training data. This intuition has motivated parameter efficient fine-tuning (PEFT), which a
saved by
- Winnie Xu
- Krish Shah
- Ratan Kaliani
- Freeman Jiang
- Emma Guo
- Karthik Suresh
- anka hu
- Hangyul Lyna Kim
- Uzay Girit
- Aaron Pham
- Dhruv Gautam
- Lydia Nottingham
related reading
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- Learning to Reason in 13 Parametersarxiv.org
- How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
- 2305.14314arxiv.org
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
- [2606.00831] Subliminal Learning is a LoRA Artifactarxiv.org
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
- 2106.09685arxiv.org