[2106.09685] LoRA: Low-Rank Adaptation of Large Language Models
Abstract:An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at this https URL.
# link_r2ad4k36mh.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20211019005713Z - Creator=LaTeX with hyperref - ModDate=D:20211019005713Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2 - Producer=pdfTeX-1.40.21 - Trapped=False ## Contents ### Page 1 LORA: LOW-RANK ADAPTATION OF LARGE LAN-GUAGE MODELSEdward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu ChenM
Explore this link on the map →saved by
related reading
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
- LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
- Recent Advances in Language Model Fine-tuningruder.io
- Understanding Large Language Modelsmagazine.sebastianraschka.com
- Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
- PEFT Method Overview [implementing Adapters in PyTorch] | Eva Korolevaxmarva.github.io
- Fine-tuning (deep learning) - Wikipediaen.wikipedia.org
- Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
- How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
- In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com