Change Data Capture with Spring Boot, Debezium, and YugabyteDB

This blog delves into the implementation of Change Data Capture (CDC) using Spring Boot and Debezium‘s embedded engine to capture and apply change events between two YugabyteDB instances, focusing on the details of this middleware-less strategy for specific use cases.

The Imperative of Real-Time Data Synchronization

Modern applications are increasingly driven by the need for immediate insights and responsiveness. This demand is particularly pronounced in microservices architectures, where data consistency across distributed services is paramount, and in analytical systems that require up-to-the-minute information for informed decision-making.

Traditional methods of data integration often rely on batch processing at scheduled intervals and can no longer keep pace with these requirements. This leads to latency, data inconsistencies, and missed opportunities for real-time action.

CDC has emerged as a critical design pattern to address this challenge as it provides a mechanism to track and react to data modifications in their source systems as they occur. CDC offers a paradigm shift from periodic bulk data transfers to a continuous stream of changes. It enables applications to stay synchronized with their underlying data sources in near real-time.

Established solutions often involve a dedicated middleware component like Apache Kafka to facilitate the transport and distribution of these change events. However, there are compelling scenarios where a more direct approach, without the added complexity and operational overhead of such middleware, can be advantageous.

Concepts and Methodologies

At its core, CDC encompasses a set of techniques designed to identify and track modifications made to data within a database. Once changes are captured, they can be replicated or processed by downstream systems, ensuring data consistency and enabling real-time responses to business events. There are several methodologies for implementing CDC, each with its own set of trade-offs.

Polling

This is one of the earliest and simplest approaches, where applications periodically query the database to identify new or modified records. This usually entails verifying timestamps or version numbers on the data. While straightforward to implement, polling can be inefficient, especially in high-throughput environments, as it places a significant load on the database.

Log-Based CDC

A more sophisticated and increasingly popular approach, log-based CDC involves reading and parsing the database transaction logs. Transaction logs record every committed change made to the database, providing a reliable and ordered stream of modifications. As it operates outside the normal data access path, this method offers very low latency and minimal impact on application performance. Debezium, a prominent open-source CDC solution, leverages this log-based approach. The inherent efficiency and accuracy of log-based CDC make it a preferred choice for applications that demand real-time data synchronization.

Snapshotting

This involves taking a full copy of the database or specific tables at certain intervals. While useful for initial data synchronization or as a fallback mechanism, snapshotting does not capture incremental changes in real-time and can be resource-intensive, especially for large databases.

Outbox Pattern

This method offers a way to ensure transactional consistency when publishing events related to data changes. In this pattern, when an application modifies data, it also inserts an event record into an “outbox” table within the same transaction. A separate process then reads these events from the outbox table and propagates them to other systems.

Your CDC methodology choice should be informed by various factors, including:

• Required latency
• Acceptable performance impact on the source database
• Need for reliability and transactional consistency
• Overall solution complexity

For real-time data synchronization with minimal overhead, log-based CDC stands out as a robust and efficient choice.

Embedded Debezium Engine

Debezium is an open-source solution specifically designed for CDC. It offers a suite of connectors that can monitor various database systems and produce a stream of change events.

While Debezium is commonly deployed within a Kafka Connect cluster, which provides a distributed and scalable environment for running connectors, it also offers an embedded engine option.

The embedded engine allows you to run a Debezium connector directly within the application’s process. In this architecture, the application itself takes on the responsibility of hosting and managing the embedded engine and the specific database connector required (e.g., for PostgreSQL or YugabyteDB). The application includes the necessary libraries and configures the engine with details about the database connection and the tables it should monitor for changes.

Each CDC approach has pros and cons, but the embedded engine approach offers some notable advantages:

1. It simplifies the overall architecture and deployment by eliminating the need for a separate Kafka and Kafka Connect infrastructure. This can lead to reduced operational overhead and costs, particularly in environments where Kafka is not already a core part of the technology stack.
2. It provides a very direct integration of change events within the application’s logic, allowing for immediate processing and custom handling of data modifications. This streamlined setup makes it an attractive option for scenarios where the full power and complexity of a distributed event streaming platform might be excessive.

However, it’s important to also acknowledge the trade-offs associated with the embedded engine:

1. Compared to a distributed Kafka Connect setup, the embedded engine typically offers lower fault tolerance and scalability. This must be implemented in the app layer to ensure scalability and fault tolerance. The resilience and performance of the CDC process are now tied to the individual application instance.
2. The application might need to handle aspects like offset management and persistence to ensure that it can resume processing from the correct point after a restart.

The decision to use the embedded engine versus Kafka Connect hinges on a careful evaluation of these factors and understanding the needs of your application and its environment.

Use Cases: Where Middleware-Less CDC With YugabyteDB Seems Appropriate

The combination of Spring Boot, Debezium’s embedded engine, and YugabyteDB offers a particularly effective solution for use cases where the simplicity and reduced operational overhead of a middleware-less approach are preferred.

CQRS (Command Query Responsibility Segregation)

The Problem: Writes need minimal indexes for speed; reads need rich indexes for performance.

Why the Embedded Approach Works: It replicates changes to a read-optimized cluster (e.g., more indexes, materialized views) while keeping the write cluster lean. It also allows field-level filtering, transformation, and custom routing to tables with different schemas or indexes.

Microservice Data Offloading

The Problem: Services need denormalized or filtered data from the primary database, but you want to avoid coupling.

Why the Embedded Approach Works: You can stream only the required data into dedicated microservice databases. The app can filter by table, columns, or business logic, so each service only gets what it needs.

Syncing with Search Engines

The Problem: You want to reflect database changes in Elasticsearch (or similar) for full-text search.

Why the Embedded Approach Works: You can stream changes from the source database to Elasticsearch with transformation logic and shape data for search (e.g., flattening, field merging) before sending it to the index.

High-Fidelity Testing and Replay

The Problem: You would like to record live production changes and replay them in staging or QA environments.

Why the Embedded Approach Works: Persist CDC logs and replays them into test systems to simulate production behavior. You have full control of when and how changes are replayed with appropriate redactions. This makes it ideal for regression and load testing.

Data Migration from PostgreSQL

The Problem: You want to migrate data from PostgreSQL to YugabyteDB.

Why the Embedded Approach Works: It allows you to continuously snapshot and stream data from PostgreSQL to YugabyteDB.

Lightweight Analytics Pipelines

The Problem: You want quick reporting but do not need a large warehouse setup (like Snowflake or Redshift).

Why the Embedded Approach Works: You can populate a reporting database optimized for reporting-style queries and enrich or aggregate the data before inserting it into the reporting layer.

While these use cases highlight the advantages of this approach, it’s important to recognize that it might not be suitable for all CDC requirements, especially those involving high scalability, robust fault tolerance across multiple independent consumers, and the need to broadcast change events to a wide range of disparate systems. In these scenarios, a more robust and decoupled solution involving a message broker like Kafka might be a better choice.

The Power of Distributed SQL with PostgreSQL DNA

The seamless integration of Spring Boot, the Debezium embedded engine, and YugabyteDB for CDC is significantly facilitated by the high degree of syntax and semantic compatibility between YugabyteDB and PostgreSQL.

This compatibility extends to the realm of CDC, simplifying both the configuration of Debezium and the use of YugabyteDB’s native logical replication. For instance, the creation of publications and replication slots in YugabyteDB closely mirrors the process in PostgreSQL. This allows you to leverage existing knowledge and documentation related to PostgreSQL logical replication when working with YugabyteDB.

It’s important to note that while compatibility is strong, there may still be specific YugabyteDB features or considerations that require additional attention. For example, YugabyteDB provides its own output plugin, yboutput, in addition to pgoutput and wal2json for logical replication, which provides unique benefits tailored to its distributed architecture. There might also be certain design considerations in YugabyteDB’s CDC implementation compared to a standard PostgreSQL setup. So, while the PostgreSQL compatibility of YugabyteDB provides a significant head start, you should consult YugabyteDB documentation for any unique configurations or limitations that might apply to your CDC implementation.

Overall, the strong alignment with PostgreSQL makes implementing CDC with YugabyteDB using both Debezium and its native capabilities a more accessible and efficient process. Beyond compatibility, YugabyteDB brings additional advantages to CDC pipelines that stem from its distributed architecture:

• High Availability: Application operation remains uninterrupted even if a node in the cluster fails. The Debezium connector with the YugabyteDB smart driver is aware of the cluster topology and will automatically re-establish traffic to available nodes.
• Transactional Guarantees: Despite being a distributed database, YugabyteDB preserves the transactional order of changes. This ensures that downstream consumers accurately reflect the state of the source data.
• Schema Evolution: YugabyteDB handles schema changes smoothly, which reduces friction when evolving your data model in real-time systems that rely on CDC.

Implementing CDC: Spring Boot, Debezium Embedded, and YugabyteDB in Action

There are a few steps to follow when implementing CDC with Spring Boot, Debezium’s embedded engine, and YugabyteDB.

First, you need to include the necessary Debezium dependencies. This includes adding the debezium-api, debezium-embedded, and the specific connector for YugabyteDB to the project’s dependency management file.

Prerequisites

Before you begin, ensure you have the following installed:

• Java Development Kit (JDK) 21 or higher: Download JDK
• Apache Maven: Download Maven
• Git: Install Git
• YugabyteDB Debezium Connector

Step 1: Clone the Repository

git clone https://github.com/srinivasa-vasu/spring-boot-cdc-stream.git
cd spring-boot-cdc-stream

Step 2: Install the YugabyteDB Debezium Connector

# download ybdb connector
wget https://github.com/yugabyte/debezium/releases/download/dz.2.5.2.yb.2024.2.3/debezium-connector-yugabytedb-dz.2.5.2.yb.2024.2.3-jar-with-dependencies.jar

# create ybdb pom
cat > debezium-yugabyte-2.5.2.Final.pom << EOF
<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                             http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>io.debezium</groupId>
    <artifactId>debezium-yugabyte</artifactId>
    <version>2.5.2.Final</version>
    <description>YugabyteDB Debezium Connector</description>
    <packaging>jar</packaging>
</project>
EOF

# install connector to the local repo
mvn install:install-file \  -Dfile=debezium-connector-yugabytedb-dz.2.5.2.yb.2024.2.3-jar-with-dependencies.jar \
  -DpomFile=debezium-yugabyte-2.5.2.Final.pom \
  -DgroupId=io.debezium \
  -DartifactId=debezium-yugabyte \
  -Dversion=2.5.2.Final \
  -Dpackaging=jar

Step 3: Set Up Source and Target Databases

• Follow the official YugabyteDB Quick Start Guide to set up a local cluster.
• Ensure that both databases are accessible and that you have the necessary credentials.

Step 4: Configure the Application

Update the application-[profile].yml file located in src/main/resources/ with producer, consumer, and datasource connection details.

Step 5: Build and Run the Application

mvn -DskipTests -Dspring-boot.run.profiles=[REPLACE_PROFILE] clean install

mvn -DskipTests -Dspring-boot.run.profiles=[REPLACE_PROFILE] spring-boot:run

Step 6: Verify CDC Functionality

To test the CDC pipeline: Insert or update data in the source database.

You can check out the complete source code on GitHub. The project is designed to be extensible, and you can customize the CDC pipeline to suit your specific needs.

Conclusion

Implementing CDC using Spring Boot, Debezium’s embedded engine, and YugabyteDB offers a compelling strategy for real-time data synchronization without the complexities of a dedicated middleware like Kafka.

This approach provides several advantages, including architecture simplicity and deployment, reduced operational overheads, direct integration of change events within the application logic, and the ability to leverage YugabyteDB’s strong compatibility with PostgreSQL.

YugabyteDB’s PostgreSQL compatibility is critical in making this approach accessible and efficient for a wide range of tools, particularly those that already support PostgreSQL as a source or target.

This middleware-less approach offers considerable benefits for specific use cases, however, your choice of CDC strategy should always be driven by the particular requirements of your application. Factors such as high scalability requirements, robust fault tolerance, broad event distribution, and integration with a diverse range of external systems will influence whether this direct integration is the most suitable solution.

For scenarios that value simplicity, low latency, and reduced operational complexity, this embedded approach provides a powerful and efficient method of achieving real-time data synchronization.

Want to learn more? Check out these additional resources:

• CDC using PostgreSQL protocol
• Debezium connector for YugabyteDB
• Debezium Engine