AWS, Cloud Computing

2 Mins Read

Change Data Capture (CDC) with Debezium on AWS

Voiced by Amazon Polly


Change Data Capture (CDC) is a crucial pattern in data engineering, capturing every change to a dataset or table and making it available for downstream systems. In this blog post, we’ll delve into implementing CDC using Debezium, an open-source CDC platform, on the AWS cloud.

What is CDC and Why Does It Matter?

Change Data Capture is essential for scenarios where:

  • Auditing historical changes to data is necessary.
  • Real-time data availability is crucial for analytical querying.
  • Event-driven architecture requires services to operate in response to changes in data.

Pioneers in Cloud Consulting & Migration Services

  • Reduced infrastructural costs
  • Accelerated application deployment
Get Started

CDC: The E and L of Your Data Pipeline

In the context of data pipelines, CDC involves two main steps:

  1. Capturing changes from the source system (E):
  • Utilizing the transaction log of the database to extract changes.
  • Incremental extraction using ordered columns.
  • Snapshot extraction for entire data.

2. Making changes available to consumers (L):

  • Extracting and loading change data into a shared location like Amazon S3.
  • Directly loading change data into a destination system.

Project Overview

Objective: Capture every change in a MySQL database and make it available for analytics.

Components of the Data Pipeline:

  • Upstream: MySQL database with user and product tables.
  • Kafka Connect Cluster: Using Debezium connector to extract data from MySQL and load it into Kafka.
  • Kafka Cluster: Making change data available for downstream consumers.
  • Data Storage: Leveraging Minio (Amazon S3 alternative) to store data generated by Debezium.
  • Data Warehouse: Utilizing duckDB to ingest data from Amazon S3 and create an SCD2 table.

AWS Environment Setup

To implement this project on AWS, consider the following AWS services:

  • AWS Cloud9: Cloud-based integrated development environment for collaborative coding.
  • Amazon S3: For scalable and secure storage of change data.
  • Amazon Kafka on MSK (Managed Streaming for Kafka): Fully managed Kafka service for building real-time data streaming applications.

Implementation Steps

  1. Environment Setup:

Set up AWS Cloud9 for a collaborative coding environment.

Ensure Docker is installed for containerized development.

  1. Debezium Configuration:

Adjust Debezium connector configurations for MySQL and Kafka.

Start Docker containers for Kafka, Zookeeper, and MySQL.

  1. Change Data Extraction:

Use Debezium to capture changes from MySQL and push them to Kafka topics.

  1. Data Loading into Amazon S3:

Set up connectors to extract data from Kafka and load it into an Amazon S3 bucket.

  1. Analysis with duckDB:

Write queries in duckDB to analyze change data and create an SCD2 dataset.

Caveats and Best Practices

  • Handling Bulk Changes: Ensure scalability of Kafka and Kafka Connect clusters for backfills or bulk changes.
  • Schema Changes: Implement mechanisms to handle schema changes gracefully.
  • Incremental Key Changes: Carefully manage incremental key changes to avoid data inconsistencies.


By mastering CDC with Debezium on AWS, you can capture and analyze changes in your data, enabling real-time analytics and ensuring data integrity. Understanding the core concepts of CDC and implementing best practices will empower you to design resilient and reliable systems.

In the ever-evolving landscape of data engineering, embracing tools like Debezium on AWS positions you at the forefront of scalable and efficient data processing.

Drop a query if you have any questions regarding CDC or Debezium and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

  • Reduced infrastructure costs
  • Timely data-driven decisions
Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services PackageCloudThat’s offerings.


1. What is CDC, and why does it matter in data engineering?

ANS: – CDC captures every change to a dataset, crucial for auditing, real-time data availability, and event-driven architectures.

2. Why use Debezium for CDC on AWS?

ANS: – Debezium, an open-source CDC platform, seamlessly integrates with Kafka, providing a reliable solution for capturing and processing change data on AWS.

WRITTEN BY Bineet Singh Kushwah

Bineet Singh Kushwah works as Associate Architect at CloudThat. His work revolves around data engineering, analytics, and machine learning projects. He is passionate about providing analytical solutions for business problems and deriving insights to enhance productivity. In a quest to learn and work with recent technologies, he spends the most time on upcoming data science trends and services in cloud platforms and keeps up with the advancements.



    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!