flâneur

Unlocking the full power of NVIDIA H100 GPUs for ML inference with TensorRT

baseten.co · 1,631 words · saved by 1 readers

NVIDIA’s H100 GPUs are the most powerful processors on the market. But running inference on ML models takes more than raw power. To get the fastest time to first token, highest tokens per second, and lowest total generation time for LLMs and models like Stable Diffusion XL, we turn to TensorRT, a model serving engine by NVIDIA. By serving models optimized with TensorRT on H100 GPUs, we unlock substantial cost savings over A100 workloads and outstanding performance benchmarks for both latency and throughput. Deploying ML models on NVIDIA H100 GPUs offers the lowest latency and highest bandwidth inference for LLMs, image generation models, and other demanding ML workloads. But getting the maximum performance from these GPUs takes more than just loading in a model and running inference. We’ve benchmarked inference for an LLM (Mistral 7B in fp16) and an image model (Stable Diffusion XL) using NVIDIA’s TensorRT and TensorRT-LLM model serving engines. Using these tools, we’ve achieved two to

TL;DR NVIDIA’s H100 GPUs are the most powerful processors on the market. But running inference on ML models takes more than raw power. To get the fastest time to first token, highest tokens per second, and lowest total generation time for LLMs and models like Stable Diffusion XL, we turn to TensorRT, a model serving engine by NVIDIA. By serving models optimized with TensorRT on H100 GPUs, we unlock substantial cost savings over A100 workloads and outstanding performance benchmarks for both latency and throughput. Deploying ML models on NVIDIA H100 GPUs offers the lowest latency and highest ban

saved by

related reading