Accelerating Generative AI with PyTorch II: GPT, Fast | PyTorch
This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment Anything over 8x using only pure, native PyTorch. In this blog we’ll focus on LLM optimization. Over the past year, generative AI use cases have exploded in popularity. Text generation has been one particularly popular area, with lots of innovation among open-source projects such as llama.cpp, vLLM, and MLC-LLM. While these projects are performant, they often come with tradeoffs in ease of use, such as requiring model conversion to specific formats or building and shipping new dependencies. This begs the question: how fast can we run transformer inference with only pure, native PyTorch? As announced during our recent PyTorch Developer Conference, th
This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment Anything over 8x using only pure, native PyTorch. In this blog we’ll focus on LLM optimization. Over the past year, generative AI use cases have exploded in popularity. Text generation has been one particularly popular area, with lots of innovation among op
saved by
related reading
- Optimizing inference · Hugging Facehuggingface.co
- Making Deep Learning go Brrrr From First Principleshorace.io
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How to make LLMs go fastvgel.me
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Speculative Decoding - philkravphilkrav.com
- Looking back at speculative decodingresearch.google