LLM Inference Performance Engineering: Best Practices | Databricks Blog

In this blog post, t

LLM Inference Performance Engineering: Best Practices | Databricks Blog Skip to main content In this blog post, the MosaicML engineering team shares best practices for how to capitalize on popular open source large language models (LLMs) for production usage. We also provide guidelines for deploying inference services built around these models to help users in their selection of models and deployment hardware. We have worked with multiple PyTorch-based backends in production; these guidelines are drawn from our experience with FasterTransformers, vLLM, NVIDIA's soon-to-be-released TensorRT-LLM

saved by

• Winnie Xu
• Amir
• Varun Shenoy
• Proby Shandilya
• Ketulkumar Polara
• Davis Brown

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Efficient LLM inferencefinbarrtimbers.substack.com
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Optimizing inference · Hugging Facehuggingface.co
• All About Transformer Inferencejax-ml.github.io
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Training LLMs with AMD MI250 GPUs and MosaicML | Databricks Blogmosaicml.com
• A guide to LLM inference and performancebaseten.co