How to achieve truly serverless GPUs
modal.com · 5,054 words · saved by 1 readers
A deep dive on Modal's deep tech for fast boots.
All posts Back Engineering May 12, 2026 • 20 minute read How we achieved truly serverless GPUs Charles Frye @charles_irl Member of Technical Staff Jonathan Belotti @jonobelotti_IO Member of Technical Staff Erik Bernhardsson @bernhardsson CEO and Founder Akshat Bubna @akshat_b CTO and Founder We are in the age of inference. Billion- to trillion-parameter neural networks are run on specialized accelerators at quadrillions of operations per second to generate media , author software , and fold proteins at massive scale. Inference workloads are more variable and less predictable than the training
saved by
related reading
- Modal: High-performance AI infrastructuremodal.com
- Modal's serverless Servers | Modal Blogmodal.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- LLM Engineer's Almanac - Workloads | Modalmodal.com
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- Low Latency and Model Training at Modalrhea24.github.io
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Reinforcement learning is an infrastructure problemmodal.com
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io