flâneur

How to design resilient and large scale data systems

blog.dataengineer.io · 1,432 words · saved by 1 readers

Whenever you’re building a data system, there’s a million things to consider. Gone are the days of just shoving everything into MongoDB and calling it “web scale.” In this newsletter, we’ll be going over the considerations you should be thinking about when building out large scale data systems. Data systems can be living and breathing in real-time like the Uber app with a data latency of milliseconds. Or they can be incredibly slow to update like the US census with a data latency of ten years! Why doesn’t Uber design their system like the US Census? I think it has something to do with the fact that ordering an Uber 10 years ahead of time isn’t classified as a “normal human behavior.” Okay, but what about the other way around? Why doesn’t the US Census design their data system like Uber? Wouldn’t having extremely up-to-date information on the population be beneficial? For the US Census, the benefit of real-time data isn’t strong enough for managing the complexity. Imagine every sing

Whenever you’re building a data system, there’s a million things to consider. Gone are the days of just shoving everything into MongoDB and calling it “web scale.” In this newsletter, we’ll be going over the considerations you should be thinking about when building out large scale data systems. Data systems can be living and breathing in real-time like the Uber app with a data latency of milliseconds. Or they can be incredibly slow to update like the US census with a data latency of ten years! Why doesn’t Uber design their system like the US Census? I think it has something to do with the…

saved by

related reading