apache flink – why the kafka lag is suddenly down to zero even though it is consuming the records?
I am using the Flink to forward the records from source to destination.
The source and sink is kafka so the dataflow is as below.
Kafka -> Flink -> Kafka
When I monitored the consumer lag in the dashboard, the pattern looks like going steadily up and then suddenly going to vertically down to almost zero in certain time as a picture below.
I know that the concept of the Kafka lag. But I don’t understand why the lag is still increasing during the more than half of hour even though the Flink consumed the records without delay.
The checkpoint of the Flink is set to 500ms, so I think that the Kafka lag should be reduced every 500ms or similar interval. But the lag is reduced in almost one hour in my case.
I understand what the lag is meaning but I am not sure that why the lag is going to down for long time even though the message is consumed without delay.
Any help will be appreciated.
Thanks.
Read more here: Source link

