flâneur

Building Venice with Apache Helix

linkedin.com · 2,421 words · saved by 1 readers

Like many internet companies, LinkedIn has faced data growth challenges. Naturally, distributed storage systems became the solution to handle larger volumes of data and queries per second (QPS). But, aside from scaling issues, the variability in access patterns also grew quickly. For example, some scenarios require no more than simple put/get operations but insist on having low latency and high availability. On the contrary, other scenarios require stronger consistency guarantees, more complex query patterns, and secondary indexes but are willing to suffer brief periods of unavailability in order to maintain these guarantees. There is no one-size-fits-all system to satisfy all needs perfectly, so LinkedIn has built specialized distributed storage systems that each offer different trade-offs in order to cover various scenarios. Now, the question that comes up is: do we build all of these systems from scratch? The answer is a resounding no! Even though scenarios vary, there are common el

Editor's note: This blog has been updated. Background Like many internet companies, LinkedIn has faced data growth challenges. Naturally, distributed storage systems became the solution to handle larger volumes of data and queries per second (QPS). But, aside from scaling issues, the variability in access patterns also grew quickly. For example, some scenarios require no more than simple put/get operations but insist on having low latency and high availability. On the contrary, other scenarios require stronger consistency guarantees, more complex query patterns, and secondary indexes but…

saved by

related reading