Debezium

Instructions on how to use WarpStream with Debezium.

Real Open Source Example

Gordon Murray has kindly created a sample repository that shows how to get started.

Tuning for Performance

If you find Debezium's performance needs to be improved, try the following things.

Update Kafka Client Configuration

Debezium leverages the Java Kafka client under the hood. It's crucial to optimize the client's configuration for your specific workload. Refer to Java Client section for recommended settings.

Important Note: Remember to add the appropriate prefixes to the configurations:

• Producer-specific settings: Use the producer. prefix (e.g., producer.linger.ms)

• Consumer-specific settings: Use the consumer. prefix (e.g., consumer.fetch.max.bytes)

These prefixes are required in Kafka Connect to ensure the settings are correctly applied within the Debezium connectors.

Update Debezium's Configuration

Each Debezium connector offers specific configurations related to the database's batching and polling that can impact performance. For instance, with the Postgres connector, consider the following optimizations:

• Batch Size: Increase the max.batch.size to process more events in a single batch.

• Queue Size: Increase the max.queue.size to allow for larger queues of events before sending them to Kafka.

• Polling Interval: Decrease the poll.interval.ms to poll for changes more frequently.

Caveat: Experiment with these settings carefully, as excessively large batches or frequent polling can also have unintended consequences, like increased resource consumption.