Accelerating Generative AI with PyTorch II: GPT, Fast | PyTorch

This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment Anything over 8x using only pure, native PyTorch. In this blog we’ll focus on LLM optimization. Over the past year, generative AI use cases have exploded in popularity. Text generation has been one particularly popular area, with lots of innovation among open-source projects such as llama.cpp, vLLM, and MLC-LLM. While these projects are performant, they often come with tradeoffs in ease of use, such as requiring model conversion to specific formats or building and shipping new dependencies. This begs the question: how fast can we run transformer inference with only pure, native PyTorch? As announced during our recent PyTorch Developer Conference, th

This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment Anything over 8x using only pure, native PyTorch. In this blog we’ll focus on LLM optimization. Over the past year, generative AI use cases have exploded in popularity. Text generation has been one particularly popular area, with lots of innovation among op

saved by

• Kevin Wang

related reading

• Optimizing inference · Hugging Facehuggingface.co
• PyTorch internals : ezyang's blogblog.ezyang.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• How To Scale Your Modeljax-ml.github.io
• How to make LLMs go fastvgel.me
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Speculative Decoding - philkravphilkrav.com
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Composer2.pdfcursor.com