LoRA vs Full Fine-tuning: An Illusion of Equivalence

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

LoRA vs Full Fine-tuning: An Illusion of Equivalence Reece Shuttleworth Jacob Andreas Antonio Torralba Pratyusha Sharma MIT CSAIL {rshuttle, jda, torralba, pratyusha}@mit.edu Abstract Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model’s weight matrices through th

saved by

• Lawrence Feng

related reading

• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
• Recovering the Pre-Fine-Tuning Weights of Generative Modelsarxiv.org
• Recent Advances in Language Model Fine-tuningruder.io
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Efficient LLM Finetuning with Unsloth | Modal Docsmodal.com
• Anatomy of a Modern Finetuning APIbenanderson.work
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io