Track every PostgreSQL data change using Debezium - DEV Community
Imagine we have a Customer Relationship Management (CRM), sales, and inventory systems. As every system's responsibility is specific, these systems only store domain-specific data in their database. Now, assume we need to generate some reports. Data is scattered in multiple databases; therefore, we must somehow gather data in one place. One idea can be to make API requests to each system and combine them later. It is definitely a bad idea to communicate systems synchronously. The good news is we can use an asynchronous Event-Driven approach. Our goal is to keep all service data in the reporting service so that it does not need to ask for related service data. To achieve this, we can publish events when data, e.g., CRM service's customer-related info, is created/updated/deleted, and the reporting service consumes the event and syncs it. Although it seems a good idea, it might be a headache when a service has a lot of tables and data gets changed using a lot of internal APIs. For simplic
Imagine we have a Customer Relationship Management (CRM), sales, and inventory systems. As every system's responsibility is specific, these systems only store domain-specific data in their database. Now, assume we need to generate some reports. Data is scattered in multiple databases; therefore, we must somehow gather data in one place. One idea can be to make API requests to each system and combine them later. It is definitely a bad idea to communicate systems synchronously. The good news is we can use an asynchronous Event-Driven approach. Our goal is to keep all service data in the reportin
Explore this link on the map →saved by
related reading
- Schema Registry in Kafka: Avro, JSON and Protobuf | by Ismael Sánchez Chaves | .Net Programming | Mediummedium.com
- GitHub - debezium/debezium: Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues. · GitHubgithub.com
- Setting up PostgreSQL for Debezium – Elephant Tamerelephanttamer.net
- Debezium and TimescaleDBdebezium.io
- Flink CDC for Postgres: Lessons Learned | sap1ens.comsap1ens.com
- PostgreSQL CDC - RisingWavedocs.risingwave.com
- Mediumlevelup.gitconnected.com
- Reliable Microservices Data Exchange With the Outbox Patterndebezium.io
- debezium-examples/postgres-toast at main · debezium/debezium-examples · GitHubgithub.com
- Debezium Releases Overviewdebezium.io
- Ingest data from self-hosted PostgreSQL | Materialize Documentationmaterialize.com
- The motivation behind CDC | solution-architecture-patternschanakaudaya.github.io