How we run GPT OSS 120B at 500+ tokens per second on NVIDIA GPUs
Day zero model performance optimization work is a mix of experimentation, bug fixing, and benchmarking guided by intuition and experience. This writeup outlines the process we followed to achieve SOTA latency and throughput for GPT OSS 120B on NVIDIA GPUs at launch with the Baseten Inference Stack. The day an open source model like OpenAI’s new gpt-oss-120b is released, we race to make the model as performant as possible for our customers. As a launch partner for OpenAI’s first open-source LLM since 2019, we wanted to give developers a great experience with the new LLMs. By the end of launch day, we were the clear leader running on NVIDIA GPUs for both latency and throughput per public data from real-world use on OpenRouter. Optimizing performance on a new model is a substantial engineering challenge. Thanks to our flexible inference stack and the collective expertise of our model performance engineering team, we are able to roll out performance improvements by the hour on new models.
TL;DR Day zero model performance optimization work is a mix of experimentation, bug fixing, and benchmarking guided by intuition and experience. This writeup outlines the process we followed to achieve SOTA latency and throughput for GPT OSS 120B on NVIDIA GPUs at launch with the Baseten Inference Stack. Update : we've made our GPT OSS implementation even faster , reaching speeds of over 650 tokens per second on Artificial Analysis and higher on OpenRouter. The day an open source model like OpenAI’s new gpt-oss-120b is released, we race to make the model as performant as possible for our custo
saved by
related reading
- Google "We Have No Moat, And Neither Does OpenAI"semianalysis.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- Let's reproduce GPT-2 (124M)youtube.com
- Noam Brown on X: "Implications of Large-Scale Test-Time Compute" / Xx.com
- Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorchpytorch.org
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- How we achieved truly serverless GPUsmodal.com
- Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRTbaseten.co