[2410.01463] Selective Aggregation for Low-Rank Adaptation in Federated Learning

Abstract:We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned $A$ and $B$ matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at this https URL.

View PDF HTML (experimental) Abstract:We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices. In doing so, we uncover that $A$ matrices are responsible for learning general knowledge, while $B$ matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices $A$ and $B$ to model the weight update, but only $A$ matrices are shared with the server for aggregation. Moreover, we delve into the relationship…

saved by

• Samuel Ratnam

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Federated learning - Wikipediaen.wikipedia.org
• LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
• 2106.09685arxiv.org
• [2310.18547] Punica: Multi-Tenant LoRA Servingarxiv.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• Learning to Reason in 13 Parametersarxiv.org
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Edge AI vs Federated Learning | Complete Overviewxenonstack.com
• What is federated learning?research.ibm.com