LoRA vs Full Fine-tuning: An Illusion of Equivalence
This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.
LoRA vs Full Fine-tuning: An Illusion of Equivalence Reece Shuttleworth Jacob Andreas Antonio Torralba Pratyusha Sharma MIT CSAIL {rshuttle, jda, torralba, pratyusha}@mit.edu Abstract Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model’s weight matrices through th
saved by
related reading
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
- 2106.09685arxiv.org
- In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
- Recovering the Pre-Fine-Tuning Weights of Generative Modelsarxiv.org
- Recent Advances in Language Model Fine-tuningruder.io
- Learning to Reason in 13 Parametersarxiv.org
- 2305.14314arxiv.org
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- [2606.00831] Subliminal Learning is a LoRA Artifactarxiv.org