Design | Apache Kafka
Motivation We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a large company might have. To do this we had to think through a fairly broad set of use cases. It would have to have high-throughput to support high volume event streams such as real-time log aggregation. It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems.
Motivation We designed Kafka to be able to act as a unified platform for handling all the real-time data feeds a large company might have. To do this we had to think through a fairly broad set of use cases. It would have to have high-throughput to support high volume event streams such as real-time log aggregation. It would need to deal gracefully with large data backlogs to be able to support periodic data loads from offline systems. It also meant the system would have to handle low-latency delivery to handle more traditional messaging use-cases. We wanted to support partitioned,…
saved by
related reading
- Mediumzicodeng.medium.com
- Why is Kafka so fast? How does it work?blog.bytebytego.com
- Exactly-once Semantics is Possible: Here's How Apache Kafka Does itconfluent.io
- Benchmarking RabbitMQ vs Kafka vs Pulsar Performanceconfluent.io
- Building Systems Using Transactions in Apache Kafka®developer.confluent.io
- When to use RabbitMQ over Kafka? - Stack Overflowstackoverflow.com
- Everything I know about good system designseangoedecke.com
- When to use RabbitMQ or Apache Kafka - CloudAMQPcloudamqp.com
- Multi-Threaded Messaging with the Apache Kafka Consumerconfluent.io
- System Design, Chapter 7: Queuescharlieinden.github.io
- learning-notes/books/designing-data-intensive-applications.md at master · keyvanakbary/learning-notes · GitHubgithub.com
- How to Choose a Message Queue?blog.bytebytego.com