flâneur

Change Data Capture Is Still an Anti-pattern. And You Still Should Use It.

streamingdata.substack.com · 1,324 words · saved by 1 readers

Change Data Capture (CDC) is a data ingestion technique used for getting data out of databases (primarily relational). There are different types of CDC, but the most powerful and efficient one is the log-based CDC. It’s usually a pull-based approach that leverages a write-ahead logging (WAL) mechanism (e.g. BinLog in MySQL or replication slots in Postgres), receives all database operations as soon as they happen and writes them to a destination, typically a streaming platform like Apache Kafka. When considered purely in the context of data ingestion, CDC may seem like a great approach. But if we zoom out and consider the role of CDC in data modelling, for example, arguably, it does more harm than good. But first, let’s explore why it’s used in the first place. Once upon a time, if you needed to get data from an application for some kind of analytics, you’d need to instrument an event. This required many things: A message broker or a streaming platform as a scalable data ingestion syste

Change Data Capture (CDC) is a data ingestion technique used for getting data out of databases (primarily relational). There are different types of CDC, but the most powerful and efficient one is the log-based CDC. It’s usually a pull-based approach that leverages a write-ahead logging (WAL) mechanism (e.g. BinLog in MySQL or replication slots in Postgres), receives all database operations as soon as they happen and writes them to a destination, typically a streaming platform like Apache Kafka. When considered purely in the context of data ingestion, CDC may seem like a great approach. But…

related reading