✳flâneur — a map of the web's best reading

Real Time Infra for Petabytes of Data a Day at Uber | LinkedIn

linkedin.com · 453 words · saved by 1 readers

As a ballpark example - with 3 trillion messages worth 3 petabytes a day, that would be 34.7GB/s and 34.7 million messages/s. That is absurdly large scale. But why do they do that? Why go the lengths to transfer & process so much data? 🤔 Well. The first question we should ask is - what is real-time data infrastructure good for? It’s good for serving use cases that require a lot of data to be answered in a timely manner. 👍 How else would you be able to do things like: match a rider and a driver? optimize the route in real time? provide a real-time adjusted ETA on when the driver will arrive & when you'll get there? know what's happening in your global business? These are simple, first order questions. But once you provide engineers with access to real-time data, a lot of ideas start to flourish. Soon, you find yourself swamped with use cases that product managers want to try out, and later productionize. According to the use cases! A naive approach would build a new data pipeline each

Top Content on LinkedIn LinkedIn respects your privacy LinkedIn and 3rd parties use essential and non-essential cookies to provide, secure, analyze and improve our Services, and to show you relevant ads (including professional and job ads ) on and off LinkedIn. Learn more in our Cookie Policy . Select Accept to consent or Reject to decline non-essential cookies for this use. You can update your choices at any time in your settings . Accept Reject Skip to main content We can’t find the page you’re looking for. The page you’re looking for may have been moved, or may no longer exist. Try going ba

Explore this link on the map →

related reading