Kafka consumer – how to recognized offset skipping/missing offsets?
nonefor the consumer and expected the consumer to stop in case it can’t find an offset
That’s what it’ll do, yes.
In practice it immediately throws an exception when the consumer gets instantiated because there’s no first/initial offset
You’ll need to actually initialize the group first, then seek it to the earliest offset. E.g.
kafka-consumer-offsets --reset-offsets --to-earliest --group connect-<name>
Something like “throw exception if offset is missing/skipped, but not on first start”?)
There’s nothing to differentiate
auto.offset.reset between “first” and “next” starts. But, you could create the connector with
consumer.override.auto.offset.reset=earliest, then wait for it to be running, then set it back to
none with a
PUT /config call. Then repeat whenever it stops running again.
JMX metric we could monitor in case a consumer is skipping messages
Not that I know of; the metrics are mostly reporting bytes processed. You’d have to additionally track how many bytes you expect it to read.
You’d need other monitoring solutions to detect log segments being deleted on the broker, and tracking those offset ranges compared to the offsets your consumer is currently reading.
Read more here: Source link