How we run GPT OSS 120B at 500+ tokens per second on NVIDIA GPUs

Day zero model performance optimization work is a mix of experimentation, bug fixing, and benchmarking guided by intuition and experience. This writeup outlines the process we followed to achieve SOTA latency and throughput for GPT OSS 120B on NVIDIA GPUs at launch with the Baseten Inference Stack. The day an open source model like OpenAI’s new gpt-oss-120b is released, we race to make the model as performant as possible for our customers. As a launch partner for OpenAI’s first open-source LLM since 2019, we wanted to give developers a great experience with the new LLMs. By the end of launch day, we were the clear leader running on NVIDIA GPUs for both latency and throughput per public data from real-world use on OpenRouter. Optimizing performance on a new model is a substantial engineering challenge. Thanks to our flexible inference stack and the collective expertise of our model performance engineering team, we are able to roll out performance improvements by the hour on new models.

TL;DR Day zero model performance optimization work is a mix of experimentation, bug fixing, and benchmarking guided by intuition and experience. This writeup outlines the process we followed to achieve SOTA latency and throughput for GPT OSS 120B on NVIDIA GPUs at launch with the Baseten Inference Stack. Update : we've made our GPT OSS implementation even faster , reaching speeds of over 650 tokens per second on Artificial Analysis and higher on OpenRouter. The day an open source model like OpenAI’s new gpt-oss-120b is released, we race to make the model as performant as possible for our custo

saved by

• Saptarshi Bhattacherya

related reading

• How To Scale Your Modeljax-ml.github.io
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Google "We Have No Moat, And Neither Does OpenAI"semianalysis.com
• Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRTbaseten.co
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorchpytorch.org
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com