flâneur

Dynamic batching | Modal Docs

modal.com · 852 words · saved by 1 readers

Modal’s @batched feature allows you to accumulate requests and process them in dynamically-sized batches, rather than one-by-one.

Dynamic batching Modal’s @batched feature allows you to accumulate requests and process them in dynamically-sized batches, rather than one-by-one. Batching increases throughput at a potential cost to latency. Batched requests can share resources and reuse work, reducing the time and cost per request. Batching is particularly useful for GPU-accelerated machine learning workloads, as GPUs are designed to maximize throughput and are frequently bottlenecked on shareable resources, like weights stored in memory. Static batching can lead to unbounded latency, as the function waits for a fixed number

saved by

related reading