[2310.18547] Punica: Multi-Tenant LoRA Serving

Abstract:Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains a new CUDA kernel design that allows batching of GPU operations for different LoRA models. This allows a GPU to hold only a single copy of the underlying pre-trained model when serving multiple, different LoRA models, significantly enhancing GPU efficiency in terms of both memory and computation. Our scheduler consolidates multi-tenant LoRA serving workloads in a shared GPU cluster. With a fixed-sized GPU cluster, our evaluations show that Punica achieves 12x higher throughput in serving multiple LoRA models compared to state-of-the-art LLM serving systems while only adding 2ms latency per token. Punica is open source at this https URL .

P UNICA : M ULTI -T ENANT L O RA S ERVING Lequn Chen * 1 Zihao Ye * 1 Yongji Wu 2 Danyang Zhuo 2 Luis Ceze 1 Arvind Krishnamurthy 1 A BSTRACT Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains…

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• How To Scale Your Modeljax-ml.github.io
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• 2106.09685arxiv.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• 📝 Guest Post: Meet LoRAX: The Open Source System that Serves 1000s of Fine-Tuned LLMs on a Single GPU*thesequence.substack.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
• KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu