apache kafka – competing consumer groups on a partition
We have certain type of messages which take a long time to process – 5% of population.
However, it is not possible to identify the message till it is actually picked up for processing by the consumer.
The objective is to clear-off/repost the subsequent messages to other partitions – the moment a ‘slow’ message is identified.
In this context, is it possible for a partition to be listened to by two consumer groups – a processor and a router.
The router front runs the processor on the partition – and clears the partition if it comes to know that the processor of that partition is ingesting a slow message at the moment(through a state flag).
ie Can a consumer group front run another on the same partition – and alter contents of the partition ,before the other one reaches the offset.
Read more here: Source link