Why Twitter Didn’t Go Down: From a Real Twitter SRE

Twitter supposedly lost around 80% of its work force. What ever the real number is, there are whole teams with out engineers on it now. Yet, the website goes on and the tweets keep coming. This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I’d like to explain my little corner of Twitter (though it wasn’t

Twitter supposedly lost around 80% of its work force. What ever the real number is, there are whole teams with out engineers on it now. Yet, the website goes on and the tweets keep coming. This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I’d like to explain my little corner of Twitter (though it wasn’t so little) and some of the work that went on that kept this thing running. For five years I was a Site Reliability Engineer(SRE) at Twitter. For four of those years I was the sole SRE for the Cache team. There was a few…

related reading

• Being oncall taught me everything - Yao Yueyaoyue.org
• Everything I know about good system designseangoedecke.com
• Production Twitter on One Machine? 100Gbps NICs and NVMe are fast - Tristan Humethume.ca
• Remains of the Dayeugenewei.com
• How to Blow Up a Timeline - Remains of the Dayeugenewei.com
• More Than DNS: The 14 hour AWS us-east-1 outage – Jonathon Belotti [thundergolfer]thundergolfer.com
• I could do that in a weekend!danluu.com
• I was laid off by Atlassianyoutube.com
• Building and operating a pretty big storage system called S3 | All Things Distributedallthingsdistributed.com
• Queues Don't Fix Overloadferd.ca
• Notes on Distributed Systems for Young Bloods – Something Similarsomethingsimilar.com
• How we reduced the cost of building Twitter at Twitter-scale by 100x – Blogblog.redplanetlabs.com