Dynamic batching | Modal Docs
modal.com · 852 words · saved by 1 readers
Modal’s @batched feature allows you to accumulate requests and process them in dynamically-sized batches, rather than one-by-one.
Dynamic batching Modal’s @batched feature allows you to accumulate requests and process them in dynamically-sized batches, rather than one-by-one. Batching increases throughput at a potential cost to latency. Batched requests can share resources and reuse work, reducing the time and cost per request. Batching is particularly useful for GPU-accelerated machine learning workloads, as GPUs are designed to maximize throughput and are frequently bottlenecked on shareable resources, like weights stored in memory. Static batching can lead to unbounded latency, as the function waits for a fixed number
saved by
related reading
- Modal's serverless Servers | Modal Blogmodal.com
- Modal: High-performance AI infrastructuremodal.com
- Low Latency and Model Training at Modalrhea24.github.io
- Introductionmodal.com
- Continuous batching from first principleshuggingface.co
- How we achieved truly serverless GPUsmodal.com
- Better MoE model inference with warp decode · Cursorcursor.com
- Lambda on hard mode: Inside Modal's web infrastructuremodal.com
- All About Transformer Inferencejax-ml.github.io
- Batch size 1 or ∞kothasuhas.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Async RL in Pure JAXdivyamakkar0.github.io