LoRA Without Regret - Thinking Machines Lab
thinkingmachines.ai · 5,825 words · saved by 37 readers
How LoRA matches full training performance more broadly than expected.
Today's leading language models contain upwards of a trillion parameters, pretrained on tens of trillions of tokens. Base model performance keeps improving with scale, as these trillions are necessary for learning and representing all the patterns in written-down human knowledge. In contrast, post-training involves smaller datasets and generally focuses on narrower domains of knowledge and ranges of behavior. It seems wasteful to use a terabit of weights to represent updates from a gigabit or megabit of training data. This intuition has motivated parameter efficient fine-tuning (PEFT), which a
saved by
- Winnie Xu
- Krish Shah
- Ratan Kaliani
- Freeman Jiang
- Emma Guo
- Karthik Suresh
- anka hu
- Hangyul Lyna Kim
- Uzay Girit
- Aaron Pham
- Dhruv Gautam
- Lydia Nottingham
related reading
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- Learning to Reason in 13 Parametersarxiv.org
- In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
- 2305.14314arxiv.org
- How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
- [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
- The Continual Learning Problemjessylin.com
- 2106.09685arxiv.org