linux – Kafka console-producer to read result of console-consumer on the fly in bash

redirect the output of [consumer topic] to [producer topic]

This is what MirrorMaker is for.

Otherwise, Kafka Streams app can be used for the same, within the same cluster.

Using this I would be able to avoid using too much disk for consumed messages

kafka-console-consumer does not write to disk. It writes to stdout. The Kafka topic data still exists on the broker’s disks.

avoid duplication

Not really? kafka-console-producer doesn’t dedupe data on its input stream.