Latency vs. Throughput in Machine Learning Pipelines

Providing practical tutorials and unconventional views on AI for physical world applications.

Contents Introduction Problem-centric View Implementation Complexity View Systems Configuration Introduction “Latency vs. Throughput in machine learning pipelines” is topic I end up discussing almost as often as how to avoid common mistakes . While many people seem to primarily think about the training side of pipelines, I’d consider the inference side a lot more interesting and challenging as there are many more trade-offs. IMHO, throughput is key on the training side - latency basically doesn’t matter although unnecessarily high latency decreases throughput of course. The thoughts presented

related reading

• Making Deep Learning go Brrrr From First Principleshorace.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• All About Transformer Inferencejax-ml.github.io
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Real-time machine learning: challenges and solutionshuyenchip.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Machine learning is going real-timehuyenchip.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• Spending Inference Time - Kevin Lukevinlu.ai