transactions – Can Flink write to multiple Kafka topics using a transactional binding?

Is there a way to sink records using KafkaSink to more than one topic ensuring transactions? The documentation does not mention anything like it.

There is this stackoverflow thread which proposes collecting the same record multiple times and routing them to different topics. My concern here is the process failing in the middle of the writing phase. Would it have a transaction binding among all sinks originated from the same source? I mean, could I end up with a committed record in one topic but not in the other one? I was expecting something like a KafkaTemplate as proposed here.

The destination topics is derived from a property of the event and a config file, think of something like a fanout with events landing into 1 or more topics. Ideally, if there is more than 1 topic, a Kafka transaction would be use to push the event.

Using out.collect multiple times would, in essense, give me the expected output. But I’m concerned about the transactional binding among those writes. I could end up with the sink record in one topic but not on the others due to a failure on sink time.

Read more here: Source link