Performance Tuning Guide — verl documentation
Below are key factors for tuning vLLM-based rollout. Before tuning, we recommend setting actor_rollout_ref.rollout.disable_log_stats=False so that rollout statistics are logged. Increase gpu_memory_utilization. For vLLM v0.7.0 and later, the vLLM instance will only use gpu_memory_utilization of the total memory. For SGLang, it’s the fraction of the free GPU memory used for static memory like model weights and KV cache. However, the remaining (1-gpu_memory_utilization) will also be used during inference. However, if model parameters and optimizer states are not offloaded, using too high a fraction can lead to OOM. A value between 0.5 and 0.7 often strikes a good balance between high throughput and avoiding OOM. Note: since the definition of gpu_memory_utilization varies across inference engines, a value that works well for one engine may cause OOM for another. Adjust max_num_seqs or max_num_batched_tokens. If the GPU cache utilization is relatively low in the log, increase max_num_seqs
Performance Tuning Guide - verl documentation Performance Tuning Guide View page source Performance Tuning Guide Last updated: 07/17/2025. Author: Guangming Sheng , Jiali Zheng In this section, we will discuss how to tune the performance of all the stages in verl, including: Rollout generation throughput. Enable use_remove_padding=True for sequence packing (i.e., data packing and remove padding). Batch size tuning for forward and backward computation Enable use_dynamic_bsz=True for higher throughput. Utilize Ulysses Sequence Parallel for Long Context Training LigerKernel for SFT performance
Explore this link on the map →related reading
- How To Scale Your Modeljax-ml.github.io
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Model Runner V2: A Modular and Faster Core for vLLM | vLLM Blogvllm.ai
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Optimizing inference · Hugging Facehuggingface.co
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev