Kafka Connect AvroConverter Union type
I have Kafka Connect connector using AvroConverter that is reading Debezium CDC Kafka topic generatet from MySQL database. The connector config looks as follows
configurationOverrides:
"key.converter": "io.confluent.connect.avro.AvroConverter"
"value.converter": "io.confluent.connect.avro.AvroConverter"
"key.converter.schema.registry.url": "https://schema-registry.com"
"value.converter.schema.registry.url": "https://schema-registry.com"
"producer.max.request.size": "20000000"
"offset.flush.timeout.ms": "20000"
"buffer.memory": "52428800"
Then I’m flattening the data by Kafka Connect transform like
connectorConfig:
"config":
"transforms": "unwrap,flatten"
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
"transforms.unwrap.drop.tombstones": "false"
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value"
"transforms.flatten.delimiter": "_"
"transforms.unwrap.delete.handling.mode": "rewrite"
"add.fields": "table,lsn"
But as a result I have complex Avro schema with Union fields like
{
"name": "is_registered",
"type": [
{
"type": "int",
"connect.default": 0,
"connect.type": "int16"
},
"null"
],
"default": 0
}
instead of simple types that are used in other fields like
{
"name": "attributes",
"type": [
"null",
"string"
],
"default": null
}
I would need to setup AvroConverter to not create such Union types for nullable fields or do some additional transformation to transform Union fields to simple fields. Is it possible? Any idea please?
Read more here: Source link