LoRA vs Full Fine-tuning: An Illusion of Equivalence

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

LoRA vs Full Fine-tuning: An Illusion of Equivalence Reece Shuttleworth Jacob Andreas Antonio Torralba Pratyusha Sharma MIT CSAIL {rshuttle, jda, torralba, pratyusha}@mit.edu Abstract Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model’s weight matrices through th

saved by

• Lawrence Feng

related reading

• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• 2106.09685arxiv.org
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• Selective Aggregation for Low-Rank Adaptation in Federated Learningarxiv.org
• [2402.05119] A Closer Look at the Limitations of Instruction Tuningarxiv.org
• 2305.14314arxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org