apache kafka – Debezium: Produce messages only upon changes to columns in column.include.list

I am running Debezium with column.include.list configured to a subset of columns on each of the observed tables of the source MySQL database. Changes to records on the source tables are being successfully published to Kafka, with each message’s values before & after only containing that subset of columns.

However, Debezium still publishes messages when changes occur on columns of the observed tables that are not in column.include.list. Those events are unnecessary to my downstream consumers, so I’d like to prevent them from being produced.

I only want changes to columns found in column.include.list to produce messages to Kafka. What is the preferred way to do this?

Using SMT Filtering seems like one way to do it—a filter that compares the before & after values of every column column.include.list (for the given table), and filters out any messages in which there is no difference. Is that a simpler way? Maybe a config for this behavior I missed in my search?

Read more here: Source link