Low-Rank Adaptation of Large Language Models (LoRA)

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

This is experimental and the API may change in the future. LoRA (Low-Rank Adaptation of Large Language Models) is a popular and lightweight training technique that significantly reduces the number of trainable parameters. It works by inserting a smaller number of new weights into the model and only these are trained. This makes training with LoRA much faster, memory-efficient, and produces smaller model weights (a few hundred MBs), which are easier to store and share. LoRA can also be combined with other training techniques like DreamBooth to speedup training. LoRA is very versatile and…

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• kudzueye/boreal-hl-v1 · Hugging Facehuggingface.co
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• GitHub - CompVis/stable-diffusion: A latent text-to-image diffusion modelgithub.com
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• Efficient LLM Finetuning with Unsloth | Modal Docsmodal.com
• Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and morenn.labml.ai
• Accelerating LLaMA with Fabric: A Comprehensive Guide to Training and Fine-Tuning LLaMA - Lightning AIlightning.ai
• Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org
• Tinkerthinkingmachines.ai
• (LoRA) Fine-Tuning FLUX.1-dev on Consumer Hardwarehuggingface.co