flâneur

More Than DNS: The 14 hour AWS us-east-1 outage

thundergolfer.com · 2,719 words · saved by 8 readers

On Monday the AWS us-east-1 region had its worst outage in over 10 years. The whole thing lasted over 16 hours and affected 140 AWS services, including, critically, EC2. SLAs were blown, an eight-figure revenue reduction will follow. Before Monday, I’d spent around 7 years in industry and never personally had production nuked by a public cloud outage. I generally regarded AWS’s reliability as excellent, industry-leading. What the hell happened? A number of smart engineers have come to this major bust-up and covered it with the blanket of a simple explanation: brain drain; race condition; it’s always DNS; the cloud is unreliable, go on-prem. You’re not going to understand software reliability if you summarize an outage of this scale in an internet comment. Frankly, I’m not even going to understand it after reading AWS’s 4000 word summary and thinking about it for hours. But I’m going to hold the hot takes and try. I wrote Modal’s internal us-east-1 incident postmortem before AWS publish

I’m on the right, biting a nail nervously. We’re in an Italian hotel because this happened on day 1 of our offsite. On Monday the AWS us-east-1 region had its worst outage in over 10 years. The whole thing lasted over 14 hours and affected 140 AWS services, including, critically, EC2. SLAs were blown, an eight-figure revenue reduction will follow. Before Monday, I’d spent around 7 years in industry and never personally had production nuked by a public cloud outage. I generally regarded AWS’s reliability as excellent, industry-leading. What the hell happened? A number of smart engineers have co

saved by

related reading