flâneur

Building Systems Using Transactions in Apache Kafka®

developer.confluent.io · 1,724 words · saved by 1 readers

How Kafka's transactions provide you with accurate, repeatable results from chains of many stream processors or microservices, connected via event streams. Apache Kafka® ships with built-in transactions, in much the same way that most relational databases do. The implementation is quite different, as we will see, but the goal is similar: to ensure that our programs create predictable and repeatable results, even when things fail. Transactions do three important things: Here, we’ll dive into transactions, looking at the problems they solve, how we should make use of them, and how they actually work under the covers. As a simple example, imagine we have an account validation service. It takes deposits in, validates them, and then sends a new message back to Kafka marking the deposit as validated. Kafka records the progress that each consumer makes by storing an offset in a special topic, called consumer_offsets. To ensure that each deposit is committed exactly once, the final two actions

Building Systems Using Transactions in Apache Kafka® Enhance your career, get your certificate as a Data Streaming Engineer | Get your Certificate Building Systems Using Transactions in Apache Kafka® How Kafka's transactions provide you with accurate, repeatable results from chains of many stream processors or microservices, connected via event streams. Apache Kafka® ships with built-in transactions, in much the same way that most relational databases do. The implementation is quite different, as we will see, but the goal is similar: to ensure that our programs create predictable and repeatabl

related reading