Low Latency and Model Training at Modal | Rhea Malik
rhea24.github.io · 2,824 words · saved by 3 readers
Reflections on building low-latency systems and agent-driven training at Modal
There's a very specific feeling that comes with shipping code that touches almost every single request in the product. You flip the flag and then just hold your breath. For the next few minutes, every request the product handles is running through code that I shipped. I sit there refreshing the latency dashboard every few seconds, half expecting to watch the line jump. It doesn't move. The graph looks exactly like it did an hour before, which was the best possible result, yet somehow the least satisfying one. This summer I interned at Modal, a serverless cloud platform for running…
saved by
related reading
- Async RL in Pure JAXdivyamakkar0.github.io
- Why Inference Matters 12x More Than Training | YC Paper Clubyoutube.com
- LLM Engineer's Almanac - Advisormodal.com
- Training Imperativesdan.io
- Reinforcement learning is an infrastructure problemmodal.com
- How GPT, Claude, and Gemini are actually trained and served – Reiner Popeyoutube.com
- Modal: High-performance AI infrastructuremodal.com
- Modal's serverless Servers | Modal Blogmodal.com
- Why We Built Our Own Background Agentbuilders.ramp.com
- Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
- Shipping at Inference-Speed | Peter Steinbergersteipete.me