flâneur

Surprising Economics of Load-Balanced Systems - Marc's Blog

brooker.co.za · 927 words · saved by 1 readers

I have a system with c servers, each of which can only handle a single concurrent request, and has no internal queuing. The servers sit behind a load balancer, which contains an infinite queue. An unlimited number of clients offer c * 0.8 requests per second to the load balancer on average. In other words, we increase the offered load linearly with c to keep the per-server load constant. Once a request arrives at a server, it takes one second to process, on average. How does the client-observed mean request time vary with c? Option A is that the mean latency decreases quickly, asymptotically approaching one second as c increases (in other words, the time spent in queue approaches zero). Option B is constant. Option C is a linear improvement, and D is a linear degradation in latency. Which curve do you, intuitively, think that the latency will follow? I asked my Twitter followers the same question, and got an interestingly mixed result: Breaking down the problem a bit will help figure o

Surprising Economics of Load-Balanced Systems - Marc's Blog Marc's Blog About Me My name is Marc Brooker. I like to build things that work, and do cool stuff. I like building big things. I also dabble in machining, welding, cooking, and skiing. I am an engineer at Amazon Web Services (AWS) in Seattle, where I work on agentic AI, especially safety and policy for agentic AI. Before that, I worked on EC2, EBS, databases, serverless, and serverless databases. All opinions are my own. Links My Publications and Videos @marcbrooker on Mastodon @MarcJBrooker on Twitter Is this blog written by AI? Surp

saved by

related reading